open-memex 0.9.0-alpha.2 → 0.9.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +2 -2
- package/CHANGELOG.md +5 -1
- package/README.md +7 -3
- package/README.zh-CN.md +2 -2
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +15 -2
- package/dist/init.js +3 -1
- package/dist/mcp.js +3 -1
- package/dist/reuse-report.js +64 -0
- package/dist/review.js +7 -0
- package/dist/store/usage.js +142 -27
- package/dist/tools/ops.js +12 -11
- package/docs/V2-DESIGN.md +3 -1
- package/docs/design/decisions.md +1 -1
- package/package.json +66 -66
- package/scripts/smoke-mcp.ts +9 -0
- package/scripts/smoke-pure.ts +64 -0
- package/scripts/test-full.ts +124 -13
- package/skills/open-memex/SKILL.md +4 -0
- package/src/cli.ts +18 -2
- package/src/init.ts +3 -1
- package/src/mcp.ts +3 -1
- package/src/reuse-report.ts +119 -0
- package/src/review.ts +6 -0
- package/src/store/usage.ts +319 -178
- package/src/tools/ops.ts +12 -10
package/AGENTS.md
CHANGED
|
@@ -35,7 +35,7 @@ npm run typecheck # tsc --noEmit — the only
|
|
|
35
35
|
npm run cli -- where | list | search "q" | add ... | forget <id> | reindex
|
|
36
36
|
npm run cli -- useful <id...> # D77-L2: explicit useful signal + cumulative counters
|
|
37
37
|
npm run cli -- <command> --help # per-command help (AI assistants discover flags this way)
|
|
38
|
-
npm run cli -- sync-status # last sync time/kind + outbox drafts + repo review states + uncommitted files
|
|
38
|
+
npm run cli -- sync-status # last sync time/kind + outbox drafts + repo review states + uncommitted files + D77 reuse report
|
|
39
39
|
npm run cli -- inventory [--format text|json|html] [--scope both] # what is remembered, in the open (D68); personal-bearing output refused inside a git worktree without --allow-personal
|
|
40
40
|
npm run cli -- audit [--scope project|personal|both] # memory health: near-dup pairs, stale actives, broken chains, personal-in-repo, expired working-state notes (read-only)
|
|
41
41
|
npm run cli -- wstate [--clear] # show/clear the session working-state note (D89 §3.6; the file it prints is editable directly)
|
|
@@ -126,7 +126,7 @@ Layout: `memories/<scope_key>/<id>.md` (YAML frontmatter + body) + `index.db` (S
|
|
|
126
126
|
|
|
127
127
|
**Markdown files are the source of truth; `index.db` is rebuildable.** `syncScope()` (in `src/store/sync.ts`) reconciles the two by mtime on plugin load and before every CLI read. When adding new frontmatter fields, update `Frontmatter` in `src/store/markdown.ts`, the `memories` table in `src/store/db.ts`, and `upsertFromFile` in `src/store/sync.ts` together, and bump the reindex path.
|
|
128
128
|
|
|
129
|
-
D77-L2 counters are the one deliberate exception: `memory_usage` (in `index.db`, owned by `src/store/usage.ts`) holds `retrieved_count` / `useful_self` / `useful_team` keyed by memory id, is created with `CREATE TABLE IF NOT EXISTS` outside the versioned schema, and is NOT dropped by `rebuildIndexSchema` — accumulated telemetry has no Markdown source to rebuild from. They never go into frontmatter and never leave the machine. Only `memory_useful` (explicit), a human approve at review, and a supersede raise the useful side; passive retrieval (search/list/injection, opt-in via `record: true`) raises only `retrieved_count` — never "fix" this by counting exposure as useful.
|
|
129
|
+
D77-L2 counters are the one deliberate exception: `memory_usage` (in `index.db`, owned by `src/store/usage.ts`) holds `retrieved_count` / `useful_self` / `useful_team` keyed by memory id, is created with `CREATE TABLE IF NOT EXISTS` outside the versioned schema, and is NOT dropped by `rebuildIndexSchema` — accumulated telemetry has no Markdown source to rebuild from. They never go into frontmatter and never leave the machine. Only `memory_useful` (explicit), a human approve at review, and a supersede raise the useful side; passive retrieval (search/list/injection, opt-in via `record: true`) raises only `retrieved_count` — never "fix" this by counting exposure as useful. D77-L3's report (`loadReuseReport` in usage.ts, pure rendering in `src/reuse-report.ts`) rides `memory_status` / `cli sync-status`: per-scope rates with the useful split shown separately (self vs team, per D77), top reused, zombies (retrieved, never useful; retracted/archived excluded from both lists), and a lifecycle tag on superseded/deprecated entries so historical reuse never reads as current guidance.
|
|
130
130
|
|
|
131
131
|
## Scope keys
|
|
132
132
|
|
package/CHANGELOG.md
CHANGED
|
@@ -9,9 +9,13 @@ the `@alpha` tag are development snapshots and are not listed individually.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
## [0.9.0-alpha.3] - 2026-11-04
|
|
13
|
+
|
|
14
|
+
- D77-L3: the reuse report — D77's third and final layer. `memory_status` (MCP) and `open-memex sync-status` now end with a **Reuse (D77)** block built from the `memory_usage` counters: per-scope `retrieved` / `useful` rates (personal + this project) with the **useful split shown separately** (`self` vs `team`, as D77 requires — the combined total drives the rate), the **top reused** memories, and **zombies** — retrieved at least once, never confirmed useful, i.e. the decay/prune candidates. Superseded/deprecated entries stay in the lists (their reuse was real) but carry a lifecycle tag like `[superseded]`, so historical data never reads to the agent as current guidance (a supersede itself ticks the old memory). The agent-facing tool result caps each list (cut disclosed, pointing at the CLI); the CLI prints them uncapped. Retracted/archived memories are excluded from both lists but still count toward the scope totals; an empty store gets an honest "no data yet" line instead of a silent section. Rendering is pure (`src/reuse-report.ts`, smoke-tested without SQLite); collection is one join query in `loadReuseReport`. The counters themselves stay local-only SQLite telemetry (D77).
|
|
15
|
+
|
|
12
16
|
## [0.9.0-alpha.2] - 2026-11-03
|
|
13
17
|
|
|
14
|
-
- D77-L2: measure usefulness by reuse (the D77 counters, layer 2). Each memory now carries local counters in a `memory_usage` table beside the rebuildable index (D77: counters live in SQLite, never in frontmatter, never off the machine): `retrieved_count` — incremented when a memory is actually returned by `memory_search`, `memory_list`, the CLI `search`/`list`, or the session-start injection — and the useful split, `useful_self` (personal memories and unpublished project drafts) vs `useful_team` (approved/published project memories). Only explicit signals raise the useful side: the new **`memory_useful`** tool (thirteenth MCP tool, seventh on the opencode plugin; also `open-memex useful <id...>`), a human approve at review (`promote` to approved), and a supersede. Passive exposure never counts as useful — the tool description, MCP handshake, READMEs, skill, and AGENTS.md all state the restraint so an agent that reads only one still behaves. `memory_usage` is created with `CREATE TABLE IF NOT EXISTS` outside the versioned schema and survives `rebuildIndexSchema` (ids come from the files); hard forget prunes the row, branch-switch removals do not. `memory_useful` returns counts plus `reuse_rate` (useful / retrieved, `n/a` when never retrieved). The D77-L3 reuse report (top reused, zero-reuse zombies, per-scope rates) remains open.
|
|
18
|
+
- D77-L2: measure usefulness by reuse (the D77 counters, layer 2). Each memory now carries local counters in a `memory_usage` table beside the rebuildable index (D77: counters live in SQLite, never in frontmatter, never off the machine): `retrieved_count` — incremented when a memory is actually returned by `memory_search`, `memory_list`, the CLI `search`/`list`, or the session-start injection — and the useful split, `useful_self` (personal memories and unpublished project drafts) vs `useful_team` (approved/published project memories). Only explicit signals raise the useful side: the new **`memory_useful`** tool (thirteenth MCP tool, seventh on the opencode plugin; also `open-memex useful <id...>`), a human approve at review (`promote` to approved, or the `--local-approve` propose shortcut), and a supersede (recorded even when the index row is momentarily missing — the Markdown file is the source of truth). Passive exposure never counts as useful — the tool description, MCP handshake, READMEs, skill, and AGENTS.md all state the restraint so an agent that reads only one still behaves. `memory_usage` is created with `CREATE TABLE IF NOT EXISTS` outside the versioned schema and survives `rebuildIndexSchema` (ids come from the files); hard forget prunes the row, branch-switch removals do not. `memory_useful` returns counts plus `reuse_rate` (useful / retrieved, `n/a` when never retrieved). The D77-L3 reuse report (top reused, zero-reuse zombies, per-scope rates) remains open.
|
|
15
19
|
|
|
16
20
|
## [0.9.0-alpha.1] - 2026-11-03
|
|
17
21
|
|
package/README.md
CHANGED
|
@@ -457,7 +457,7 @@ they only matter once you share memories through Git.
|
|
|
457
457
|
| ● `memory_forget` | Delete a memory by id (`soft=true` hides it instead — retracted, one-way) |
|
|
458
458
|
| ● `memory_workingstate` | Read/write/clear the per-project working-state note (§3.6, D89) — resumes an interrupted task; scratch state, never a memory |
|
|
459
459
|
| ● `memory_useful` | Mark a memory as useful (D77) — explicit confirmation only; exposure is counted separately as `retrieved_count` |
|
|
460
|
-
| `memory_status` | Show the sync queue
|
|
460
|
+
| `memory_status` | Show the sync queue (outbox drafts, repo review states, uncommitted files) plus the D77 reuse report |
|
|
461
461
|
| `memory_submit` | Move named drafts into the repo memory dir (local branch + commit) |
|
|
462
462
|
| `memory_propose` | Copy personal memories into the project scope as review candidates |
|
|
463
463
|
| `memory_promote` | Advance `proposed → approved → published` (or reject / resubmit) |
|
|
@@ -478,8 +478,12 @@ confirms it paid off, a human approval at review, or a supersede. Passive
|
|
|
478
478
|
exposure never counts as useful — a memory that merely showed up in results
|
|
479
479
|
has shown nothing yet. Same-side CLI surface: `open-memex useful <id...>`.
|
|
480
480
|
Counters live in the local SQLite index only (never in the Markdown files,
|
|
481
|
-
never off this machine)
|
|
482
|
-
per-scope rates
|
|
481
|
+
never off this machine). The report rides `memory_status` (and
|
|
482
|
+
`open-memex sync-status`): per-scope retrieved/useful rates with the self/team
|
|
483
|
+
split shown separately, the most-reused memories, and the "zombies" —
|
|
484
|
+
retrieved but never confirmed useful, i.e. the decay/prune candidates
|
|
485
|
+
(D77-L3). Superseded/deprecated entries stay visible as reuse history but
|
|
486
|
+
carry a lifecycle tag, never presented as current guidance.
|
|
483
487
|
|
|
484
488
|
## Memory types
|
|
485
489
|
|
package/README.zh-CN.md
CHANGED
|
@@ -297,7 +297,7 @@ MCP 服务器暴露十三个工具;opencode 原生插件暴露其中七个核
|
|
|
297
297
|
| ● `memory_forget` | 按 id 删除一条记忆(`soft=true` 改为隐藏——retracted,单向) |
|
|
298
298
|
| ● `memory_workingstate` | 读/写/清除每项目的工作状态笔记(§3.6,D89)——让被中断的任务得以续接;只是草稿状态,永远不会成为记忆 |
|
|
299
299
|
| ● `memory_useful` | 确认一条记忆有用(D77)——只认显式确认;被动的搜索/列表出现只计入 `retrieved_count`,绝不算有用 |
|
|
300
|
-
| `memory_status` |
|
|
300
|
+
| `memory_status` | 显示同步队列(outbox 草稿、仓库评审状态、未提交文件),并附带 D77 复用报告 |
|
|
301
301
|
| `memory_submit` | 把指定的草稿移进仓库记忆目录(本地分支 + 提交) |
|
|
302
302
|
| `memory_propose` | 把个人记忆复制到项目作用域,作为评审候选 |
|
|
303
303
|
| `memory_promote` | 推进 `proposed → approved → published`(或拒绝 / 重新提交) |
|
|
@@ -306,7 +306,7 @@ MCP 服务器暴露十三个工具;opencode 原生插件暴露其中七个核
|
|
|
306
306
|
|
|
307
307
|
所以 MCP 接入的编辑器永远是全套;opencode 插件主攻捕获与回想,工作流类的事走 CLI 或 MCP 接入的编辑器。
|
|
308
308
|
|
|
309
|
-
**有用性靠复用来衡量,不靠存储量(D77)。** 每条记忆带本地计数器:`retrieved_count`——它真正被递交给助手的次数(搜索结果、清单、会话开始注入);有用计数则拆成 `useful_self`(个人记忆、未评审的项目草稿)和 `useful_team`(approved/published 的项目记忆)。只有显式信号会抬升有用侧:`memory_useful`(人确认它帮上忙时)、评审中的人为 approve、或一次 supersede。被动曝光永远不算有用——记忆只是出现在结果里,还不能说明任何事。同侧的人用入口:`open-memex useful <id...>`。计数器只存在本地 SQLite 索引里(不进 Markdown
|
|
309
|
+
**有用性靠复用来衡量,不靠存储量(D77)。** 每条记忆带本地计数器:`retrieved_count`——它真正被递交给助手的次数(搜索结果、清单、会话开始注入);有用计数则拆成 `useful_self`(个人记忆、未评审的项目草稿)和 `useful_team`(approved/published 的项目记忆)。只有显式信号会抬升有用侧:`memory_useful`(人确认它帮上忙时)、评审中的人为 approve、或一次 supersede。被动曝光永远不算有用——记忆只是出现在结果里,还不能说明任何事。同侧的人用入口:`open-memex useful <id...>`。计数器只存在本地 SQLite 索引里(不进 Markdown 文件、不出这台机器)。复用报告挂在 `memory_status`(以及 `open-memex sync-status`)上:各作用域的 retrieved/useful 比率(self/team 分开显示)、复用最多的记忆,以及"僵尸"——被检索过却从未被确认有用、即衰减/清理候选项(D77-L3)。superseded/deprecated 的记忆仍作为复用历史出现,但带生命周期标记,不会被当成当前指引。
|
|
310
310
|
|
|
311
311
|
## 记忆类型
|
|
312
312
|
|
package/dist/.build-stamp
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
507da63044e3c491b1ec5b7f077ccfbc492c986775ce97b77f172e5395fccca3
|
package/dist/cli.js
CHANGED
|
@@ -10,7 +10,8 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
|
|
|
10
10
|
import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.js";
|
|
11
11
|
import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.js";
|
|
12
12
|
import { search, list, countList, hitStateLabel } from "./retrieve/search.js";
|
|
13
|
-
import { recordUseful, getUsage, reuseRate, usefulCount, usefulKindFor, deleteUsage } from "./store/usage.js";
|
|
13
|
+
import { recordUseful, getUsage, reuseRate, usefulCount, usefulKindFor, deleteUsage, loadReuseReport } from "./store/usage.js";
|
|
14
|
+
import { formatReuseReport } from "./reuse-report.js";
|
|
14
15
|
import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.js";
|
|
15
16
|
import { formatInventoryLine, truncationNote } from "./retrieve/display.js";
|
|
16
17
|
import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, normalizeAliases, } from "./store/markdown.js";
|
|
@@ -206,7 +207,9 @@ Examples:
|
|
|
206
207
|
open-memex resolve 01ABC`,
|
|
207
208
|
"sync-status": `Show the project memory sync pipeline: when the index last synced
|
|
208
209
|
and what triggered it, drafts waiting in the outbox (appdata), memories in the
|
|
209
|
-
repo awaiting review or published, and repo files not yet committed.
|
|
210
|
+
repo awaiting review or published, and repo files not yet committed. The
|
|
211
|
+
D77 reuse report follows: per-scope retrieved/useful rates, most-reused
|
|
212
|
+
memories, and "zombies" (retrieved but never confirmed useful).
|
|
210
213
|
|
|
211
214
|
Usage: open-memex sync-status
|
|
212
215
|
|
|
@@ -1400,6 +1403,16 @@ async function main() {
|
|
|
1400
1403
|
syncScope(PERSONAL_SCOPE.key, "cli");
|
|
1401
1404
|
try {
|
|
1402
1405
|
console.log(formatSyncStatus(getSyncStatus(), { cap: Number.POSITIVE_INFINITY }));
|
|
1406
|
+
// D77-L3: same reuse block as the memory_status tool, uncapped (a
|
|
1407
|
+
// human asked; the CLI is where the full list lives).
|
|
1408
|
+
const reuse = loadReuseReport([
|
|
1409
|
+
{ scopeKey: project.key, label: "this project" },
|
|
1410
|
+
{ scopeKey: PERSONAL_SCOPE.key, label: "personal" },
|
|
1411
|
+
], { topLimit: Number.POSITIVE_INFINITY, zombieLimit: Number.POSITIVE_INFINITY });
|
|
1412
|
+
console.log("");
|
|
1413
|
+
for (const line of formatReuseReport(reuse, { entryCap: Number.POSITIVE_INFINITY })) {
|
|
1414
|
+
console.log(line);
|
|
1415
|
+
}
|
|
1403
1416
|
}
|
|
1404
1417
|
catch (e) {
|
|
1405
1418
|
console.error(`sync-status failed: ${e.message}`);
|
package/dist/init.js
CHANGED
|
@@ -107,7 +107,9 @@ yet. Syncing them into the repo for review is an explicit, user-approved step:
|
|
|
107
107
|
- When the user says "sync memory" (or "同步记忆"), run the sync
|
|
108
108
|
flow above: call \`memory_status\`, summarize the outbox drafts, and ask which
|
|
109
109
|
ones to sync. ALWAYS use the \`memory_status\` tool for this — never browse
|
|
110
|
-
the memory data directory directly.
|
|
110
|
+
the memory data directory directly. That tool also carries the D77 reuse
|
|
111
|
+
report (per-scope rates, most-reused memories, and "zombies"): surface it
|
|
112
|
+
when the user asks what memory is actually being reused.
|
|
111
113
|
- When the user approves, call \`memory_submit\` with the approved ids. It
|
|
112
114
|
copies the drafts into the current project's \`.ai/open-memex/\` directory as
|
|
113
115
|
\`proposed\` and commits locally on the CURRENT branch. It NEVER creates a
|
package/dist/mcp.js
CHANGED
|
@@ -67,7 +67,9 @@ them by these full names.
|
|
|
67
67
|
- When the user says "sync memory" (or "同步记忆"), call memory_status,
|
|
68
68
|
summarize the outbox drafts (one line each), and ask which ones to sync.
|
|
69
69
|
ALWAYS use the memory_status tool for this — never browse the memory data
|
|
70
|
-
directory directly.
|
|
70
|
+
directory directly. memory_status also carries the D77 reuse report
|
|
71
|
+
(per-scope rates, most-reused, and never-used-but-retrieved "zombies") —
|
|
72
|
+
surface it when the user asks what memory is actually being reused.
|
|
71
73
|
- After memory_submit, ask ONE follow-up: "want me to create a branch + push +
|
|
72
74
|
open the PR, or will you handle it yourself?" NEVER create branches, push, or
|
|
73
75
|
open PRs without the user's explicit approval. A "yes, you do it" covers the
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D77-L3: the reuse report — pure types + rendering, no SQLite import (the
|
|
3
|
+
* data comes from `loadReuseReport` in store/usage.ts). Split like
|
|
4
|
+
* weights.ts so smoke-pure can test the formatting without a database.
|
|
5
|
+
*
|
|
6
|
+
* Report rules (D77):
|
|
7
|
+
* - per-scope reuse rate = useful / retrieved over tracked memories
|
|
8
|
+
* (counters exist per memory; scopes are personal vs the current project);
|
|
9
|
+
* - the useful side is shown split into self vs team — D77 requires reporting
|
|
10
|
+
* them separately so the team-knowledge claim stays honest (the combined
|
|
11
|
+
* total still drives ranking and the rate);
|
|
12
|
+
* - "top reused" = highest useful, then highest exposure;
|
|
13
|
+
* - "zombies" = retrieved at least once, never confirmed useful — the
|
|
14
|
+
* decay/prune candidates the D77 text calls out;
|
|
15
|
+
* - retracted/archived memories are excluded from the two lists (their
|
|
16
|
+
* counters still count toward the scope totals — history is history);
|
|
17
|
+
* - superseded/deprecated entries stay in the lists (their reuse is real)
|
|
18
|
+
* but carry a lifecycle tag, so historical data never reads as current
|
|
19
|
+
* guidance — supersede itself ticks the OLD memory's useful counter;
|
|
20
|
+
* - lists are capped and the cut is disclosed (D68 honesty rule).
|
|
21
|
+
*/
|
|
22
|
+
function rateText(rate) {
|
|
23
|
+
return rate === null ? "n/a" : `${Math.round(rate * 100)}%`;
|
|
24
|
+
}
|
|
25
|
+
/**
|
|
26
|
+
* Render the report as plain lines. Always emits the "Reuse (D77" header, so
|
|
27
|
+
* the section exists (with an honest empty-state line) even before any
|
|
28
|
+
* retrieval has been recorded — discoverability over silence.
|
|
29
|
+
*/
|
|
30
|
+
export function formatReuseReport(report, opts = {}) {
|
|
31
|
+
const cap = opts.entryCap ?? 5;
|
|
32
|
+
const lines = ["Reuse (D77 — local counters, never leave this machine):"];
|
|
33
|
+
if (report.tracked === 0) {
|
|
34
|
+
lines.push(" no data yet — search/list/session-start injection record exposure; memory_useful records usefulness.");
|
|
35
|
+
return lines;
|
|
36
|
+
}
|
|
37
|
+
for (const s of report.scopes) {
|
|
38
|
+
if (s.tracked === 0)
|
|
39
|
+
continue;
|
|
40
|
+
lines.push(` ${s.label}: retrieved ${s.retrieved}, useful ${s.useful} (self ${s.usefulSelf}, team ${s.usefulTeam}) — ${rateText(s.rate)} over ${s.tracked} memor${s.tracked === 1 ? "y" : "ies"}`);
|
|
41
|
+
}
|
|
42
|
+
// Only disambiguate each entry's scope when both scopes have data — the
|
|
43
|
+
// single-scope case would just repeat the same tag on every line.
|
|
44
|
+
const multi = report.scopes.filter((s) => s.tracked > 0).length > 1;
|
|
45
|
+
const tag = (e) => (multi ? `[${e.scopeLabel}/${e.type}]` : `[${e.type}]`);
|
|
46
|
+
// Lifecycle tag: superseded/deprecated entries are historical reuse, not
|
|
47
|
+
// current guidance (a supersede ticks the old memory, so it can rank high).
|
|
48
|
+
const life = (e) => (e.status && e.status !== "active" ? ` [${e.status}]` : "");
|
|
49
|
+
lines.push(` top reused: ${report.topTotal}`);
|
|
50
|
+
for (const e of report.top.slice(0, cap)) {
|
|
51
|
+
lines.push(` · ${e.id} ${tag(e)}${life(e)} retrieved=${e.retrieved} useful=${e.useful} (self ${e.usefulSelf}, team ${e.usefulTeam}) — ${e.snippet}`);
|
|
52
|
+
}
|
|
53
|
+
if (report.topTotal > cap) {
|
|
54
|
+
lines.push(` … and ${report.topTotal - cap} more (full list: open-memex sync-status)`);
|
|
55
|
+
}
|
|
56
|
+
lines.push(` zombies (retrieved, never confirmed useful): ${report.zombieTotal}`);
|
|
57
|
+
for (const e of report.zombies.slice(0, cap)) {
|
|
58
|
+
lines.push(` · ${e.id} ${tag(e)}${life(e)} retrieved=${e.retrieved} — ${e.snippet}`);
|
|
59
|
+
}
|
|
60
|
+
if (report.zombieTotal > cap) {
|
|
61
|
+
lines.push(` … and ${report.zombieTotal - cap} more (full list: open-memex sync-status)`);
|
|
62
|
+
}
|
|
63
|
+
return lines;
|
|
64
|
+
}
|
package/dist/review.js
CHANGED
|
@@ -109,6 +109,13 @@ function copyPersonalToProject(sourceId, srcFm, srcBody, opts = {}) {
|
|
|
109
109
|
};
|
|
110
110
|
const { filePath } = writeMemoryFile(fm, srcBody);
|
|
111
111
|
upsertFromFile({ fm, body: srcBody, filePath, mtimeMs: Date.now() });
|
|
112
|
+
// D77-L2: a local approval is still a human approval at review, so it must
|
|
113
|
+
// record the same useful_team tick the promote path records. It bypasses
|
|
114
|
+
// promoteMemory by design (solo-dev shortcut) and submit preserves the
|
|
115
|
+
// approved state, so no later step would otherwise count it — and the
|
|
116
|
+
// later `approved → published` promote stays non-counting for both paths.
|
|
117
|
+
if (targetState === "approved")
|
|
118
|
+
recordUseful(fm.id, "team", "promote");
|
|
112
119
|
return { id: fm.id, filePath, reviewState: fm.review_state };
|
|
113
120
|
}
|
|
114
121
|
/**
|
package/dist/store/usage.js
CHANGED
|
@@ -35,23 +35,23 @@ export function usefulKindFor(scope, reviewState) {
|
|
|
35
35
|
return "team";
|
|
36
36
|
return "self";
|
|
37
37
|
}
|
|
38
|
-
const INSERT_RETRIEVED_SQL = `
|
|
39
|
-
INSERT INTO memory_usage (id, retrieved_count, useful_self, useful_team, last_retrieved_at, last_useful_at, updated_at)
|
|
40
|
-
VALUES (?, 1, 0, 0, ?, NULL, ?)
|
|
41
|
-
ON CONFLICT(id) DO UPDATE SET
|
|
42
|
-
retrieved_count = memory_usage.retrieved_count + 1,
|
|
43
|
-
last_retrieved_at = excluded.last_retrieved_at,
|
|
44
|
-
updated_at = excluded.updated_at
|
|
38
|
+
const INSERT_RETRIEVED_SQL = `
|
|
39
|
+
INSERT INTO memory_usage (id, retrieved_count, useful_self, useful_team, last_retrieved_at, last_useful_at, updated_at)
|
|
40
|
+
VALUES (?, 1, 0, 0, ?, NULL, ?)
|
|
41
|
+
ON CONFLICT(id) DO UPDATE SET
|
|
42
|
+
retrieved_count = memory_usage.retrieved_count + 1,
|
|
43
|
+
last_retrieved_at = excluded.last_retrieved_at,
|
|
44
|
+
updated_at = excluded.updated_at
|
|
45
45
|
`;
|
|
46
46
|
function insertUsefulSql(self) {
|
|
47
47
|
const col = self ? "useful_self" : "useful_team";
|
|
48
|
-
return `
|
|
49
|
-
INSERT INTO memory_usage (id, retrieved_count, useful_self, useful_team, last_retrieved_at, last_useful_at, updated_at)
|
|
50
|
-
VALUES (?, 0, ${self ? 1 : 0}, ${self ? 0 : 1}, '', ?, ?)
|
|
51
|
-
ON CONFLICT(id) DO UPDATE SET
|
|
52
|
-
${col} = memory_usage.${col} + 1,
|
|
53
|
-
last_useful_at = excluded.last_useful_at,
|
|
54
|
-
updated_at = excluded.updated_at
|
|
48
|
+
return `
|
|
49
|
+
INSERT INTO memory_usage (id, retrieved_count, useful_self, useful_team, last_retrieved_at, last_useful_at, updated_at)
|
|
50
|
+
VALUES (?, 0, ${self ? 1 : 0}, ${self ? 0 : 1}, '', ?, ?)
|
|
51
|
+
ON CONFLICT(id) DO UPDATE SET
|
|
52
|
+
${col} = memory_usage.${col} + 1,
|
|
53
|
+
last_useful_at = excluded.last_useful_at,
|
|
54
|
+
updated_at = excluded.updated_at
|
|
55
55
|
`;
|
|
56
56
|
}
|
|
57
57
|
function rfc(ms) {
|
|
@@ -82,23 +82,27 @@ export function recordRetrieved(ids, atMs = Date.now()) {
|
|
|
82
82
|
}
|
|
83
83
|
export const VIA_VALUES = ["tool", "supersede", "promote"];
|
|
84
84
|
/**
|
|
85
|
-
* Count one explicit useful signal.
|
|
86
|
-
*
|
|
87
|
-
*
|
|
85
|
+
* Count one explicit useful signal. The upsert is a single statement, so it
|
|
86
|
+
* is atomic on both drivers — there is deliberately no separate existence
|
|
87
|
+
* check: a memory can be valid on disk while its index row is momentarily
|
|
88
|
+
* missing (schema rebuild, or `findMemoryFile`'s file-scan fallback), and a
|
|
89
|
+
* guard keyed on `memories` would silently drop a real supersede/approve
|
|
90
|
+
* signal. The explicit surfaces (`memory_useful`, CLI `useful`) do their own
|
|
91
|
+
* not-found check before calling, so an agent's typo id never lands here.
|
|
92
|
+
* A concurrent hard forget can still leave an orphaned row; `listUsage`
|
|
93
|
+
* inner-joins `memories`, so orphans never surface in reports.
|
|
88
94
|
*/
|
|
89
95
|
export function recordUseful(id, kind, _via, atMs = Date.now()) {
|
|
90
96
|
const d = db();
|
|
91
|
-
const mem = d.prepare(`SELECT id FROM memories WHERE id = ?`).get(id);
|
|
92
|
-
if (!mem)
|
|
93
|
-
return null;
|
|
94
97
|
const at = rfc(atMs);
|
|
95
98
|
d.prepare(insertUsefulSql(kind === "self")).run(id, at, at);
|
|
99
|
+
// getUsage cannot miss the row just upserted.
|
|
96
100
|
return getUsage(id);
|
|
97
101
|
}
|
|
98
102
|
export function getUsage(id) {
|
|
99
103
|
const row = db()
|
|
100
|
-
.prepare(`SELECT id, retrieved_count, useful_self, useful_team,
|
|
101
|
-
last_retrieved_at, last_useful_at, updated_at
|
|
104
|
+
.prepare(`SELECT id, retrieved_count, useful_self, useful_team,
|
|
105
|
+
last_retrieved_at, last_useful_at, updated_at
|
|
102
106
|
FROM memory_usage WHERE id = ?`)
|
|
103
107
|
.get(id);
|
|
104
108
|
if (!row)
|
|
@@ -122,12 +126,12 @@ export function reuseRate(u) {
|
|
|
122
126
|
/** D77-L3's selection helpers ride the same table scan for free. */
|
|
123
127
|
export function listUsage(scopeKey) {
|
|
124
128
|
const sql = scopeKey
|
|
125
|
-
? `SELECT u.id, u.retrieved_count, u.useful_self, u.useful_team,
|
|
126
|
-
u.last_retrieved_at, u.last_useful_at, u.updated_at
|
|
127
|
-
FROM memory_usage u JOIN memories m ON m.id = u.id
|
|
129
|
+
? `SELECT u.id, u.retrieved_count, u.useful_self, u.useful_team,
|
|
130
|
+
u.last_retrieved_at, u.last_useful_at, u.updated_at
|
|
131
|
+
FROM memory_usage u JOIN memories m ON m.id = u.id
|
|
128
132
|
WHERE m.scope_key = ?`
|
|
129
|
-
: `SELECT u.id, u.retrieved_count, u.useful_self, u.useful_team,
|
|
130
|
-
u.last_retrieved_at, u.last_useful_at, u.updated_at
|
|
133
|
+
: `SELECT u.id, u.retrieved_count, u.useful_self, u.useful_team,
|
|
134
|
+
u.last_retrieved_at, u.last_useful_at, u.updated_at
|
|
131
135
|
FROM memory_usage u`;
|
|
132
136
|
const args = scopeKey ? [scopeKey] : [];
|
|
133
137
|
const rows = db().prepare(sql).all(...args);
|
|
@@ -138,6 +142,117 @@ export function listUsage(scopeKey) {
|
|
|
138
142
|
}
|
|
139
143
|
return rows;
|
|
140
144
|
}
|
|
145
|
+
function oneLine(s, max = 120) {
|
|
146
|
+
const t = s.replace(/\s+/g, " ").trim();
|
|
147
|
+
return t.length > max ? t.slice(0, max - 1) + "…" : t;
|
|
148
|
+
}
|
|
149
|
+
/**
|
|
150
|
+
* D77-L3: assemble the reuse report for the given scopes (personal and the
|
|
151
|
+
* current project on every surface). One join query, computed in JS — the
|
|
152
|
+
* store is local and the row count is a corpus, not a warehouse.
|
|
153
|
+
*
|
|
154
|
+
* Scope totals include every row with telemetry (retracted/archived too:
|
|
155
|
+
* their exposure was real while it lasted) and keep the useful split —
|
|
156
|
+
* D77 requires self vs team reported separately. The two actionable lists —
|
|
157
|
+
* top reused and zombies — exclude retracted/archived, because neither
|
|
158
|
+
* pruning nor celebrating a withdrawn memory helps; superseded/deprecated
|
|
159
|
+
* rows stay (their reuse is real) and carry their status so the render can
|
|
160
|
+
* mark them as history rather than current guidance.
|
|
161
|
+
*/
|
|
162
|
+
export function loadReuseReport(scopes, opts = {}) {
|
|
163
|
+
const topLimit = opts.topLimit ?? 5;
|
|
164
|
+
const zombieLimit = opts.zombieLimit ?? 5;
|
|
165
|
+
const labels = new Map(scopes.map((s) => [s.scopeKey, s.label]));
|
|
166
|
+
const empty = {
|
|
167
|
+
scopes: scopes.map((s) => ({
|
|
168
|
+
label: s.label,
|
|
169
|
+
scopeKey: s.scopeKey,
|
|
170
|
+
tracked: 0,
|
|
171
|
+
retrieved: 0,
|
|
172
|
+
useful: 0,
|
|
173
|
+
usefulSelf: 0,
|
|
174
|
+
usefulTeam: 0,
|
|
175
|
+
rate: null,
|
|
176
|
+
})),
|
|
177
|
+
top: [],
|
|
178
|
+
topTotal: 0,
|
|
179
|
+
zombies: [],
|
|
180
|
+
zombieTotal: 0,
|
|
181
|
+
tracked: 0,
|
|
182
|
+
};
|
|
183
|
+
if (scopes.length === 0)
|
|
184
|
+
return empty;
|
|
185
|
+
const keys = scopes.map((s) => s.scopeKey);
|
|
186
|
+
const rows = db()
|
|
187
|
+
.prepare(`SELECT u.id, u.retrieved_count, u.useful_self, u.useful_team,
|
|
188
|
+
m.scope_key, m.type, m.review_state, m.status,
|
|
189
|
+
substr(m.content, 1, 200) AS content
|
|
190
|
+
FROM memory_usage u JOIN memories m ON m.id = u.id
|
|
191
|
+
WHERE m.scope_key IN (${keys.map(() => "?").join(",")})`)
|
|
192
|
+
.all(...keys);
|
|
193
|
+
const byScope = new Map();
|
|
194
|
+
for (const s of empty.scopes)
|
|
195
|
+
byScope.set(s.scopeKey, s);
|
|
196
|
+
const tracked = [];
|
|
197
|
+
for (const r of rows) {
|
|
198
|
+
const retrieved = Number(r.retrieved_count) || 0;
|
|
199
|
+
const usefulSelf = Number(r.useful_self) || 0;
|
|
200
|
+
const usefulTeam = Number(r.useful_team) || 0;
|
|
201
|
+
const useful = usefulSelf + usefulTeam;
|
|
202
|
+
const stat = byScope.get(r.scope_key);
|
|
203
|
+
if (stat) {
|
|
204
|
+
stat.tracked++;
|
|
205
|
+
stat.retrieved += retrieved;
|
|
206
|
+
stat.useful += useful;
|
|
207
|
+
stat.usefulSelf += usefulSelf;
|
|
208
|
+
stat.usefulTeam += usefulTeam;
|
|
209
|
+
}
|
|
210
|
+
tracked.push({
|
|
211
|
+
id: r.id,
|
|
212
|
+
scopeKey: r.scope_key,
|
|
213
|
+
scopeLabel: labels.get(r.scope_key) ?? r.scope_key,
|
|
214
|
+
type: r.type,
|
|
215
|
+
reviewState: r.review_state ?? "draft",
|
|
216
|
+
status: r.status,
|
|
217
|
+
retrieved,
|
|
218
|
+
useful,
|
|
219
|
+
usefulSelf,
|
|
220
|
+
usefulTeam,
|
|
221
|
+
snippet: oneLine(r.content ?? ""),
|
|
222
|
+
});
|
|
223
|
+
}
|
|
224
|
+
for (const stat of byScope.values()) {
|
|
225
|
+
stat.rate = stat.retrieved > 0 ? stat.useful / stat.retrieved : null;
|
|
226
|
+
}
|
|
227
|
+
const actionable = tracked.filter((e) => e.status !== "retracted" && e.status !== "archived");
|
|
228
|
+
const top = actionable
|
|
229
|
+
.filter((e) => e.useful > 0)
|
|
230
|
+
.sort((a, b) => b.useful - a.useful || b.retrieved - a.retrieved || a.id.localeCompare(b.id));
|
|
231
|
+
const zombies = actionable
|
|
232
|
+
.filter((e) => e.retrieved > 0 && e.useful === 0)
|
|
233
|
+
.sort((a, b) => b.retrieved - a.retrieved || a.id.localeCompare(b.id));
|
|
234
|
+
const strip = (e) => ({
|
|
235
|
+
id: e.id,
|
|
236
|
+
scopeKey: e.scopeKey,
|
|
237
|
+
scopeLabel: e.scopeLabel,
|
|
238
|
+
type: e.type,
|
|
239
|
+
reviewState: e.reviewState,
|
|
240
|
+
status: e.status,
|
|
241
|
+
retrieved: e.retrieved,
|
|
242
|
+
useful: e.useful,
|
|
243
|
+
usefulSelf: e.usefulSelf,
|
|
244
|
+
usefulTeam: e.usefulTeam,
|
|
245
|
+
snippet: e.snippet,
|
|
246
|
+
});
|
|
247
|
+
return {
|
|
248
|
+
scopes: empty.scopes,
|
|
249
|
+
top: top.slice(0, topLimit).map(strip),
|
|
250
|
+
topTotal: top.length,
|
|
251
|
+
zombies: zombies.slice(0, zombieLimit).map(strip),
|
|
252
|
+
zombieTotal: zombies.length,
|
|
253
|
+
tracked: tracked.length,
|
|
254
|
+
};
|
|
255
|
+
}
|
|
141
256
|
/**
|
|
142
257
|
* Drop one memory's telemetry. ONLY the hard-forget path calls this (the
|
|
143
258
|
* memory is gone for good); plain index removal — which fires on a branch
|
package/dist/tools/ops.js
CHANGED
|
@@ -15,7 +15,8 @@ import { writeMemoryFile, readMemoryFile, ulid, msToRfc3339, MEMORY_TYPE_TAXONOM
|
|
|
15
15
|
import { upsertFromFile, deleteFromIndex } from "../store/sync.js";
|
|
16
16
|
import { findDuplicates, findTurnEcho, supersede, setStatus } from "../store/lifecycle.js";
|
|
17
17
|
import { TURN_ECHO_WINDOW_MS } from "../capture/handoff.js";
|
|
18
|
-
import { getUsage, recordUseful, reuseRate, usefulKindFor, usefulCount, deleteUsage, } from "../store/usage.js";
|
|
18
|
+
import { getUsage, recordUseful, reuseRate, usefulKindFor, usefulCount, deleteUsage, loadReuseReport, } from "../store/usage.js";
|
|
19
|
+
import { formatReuseReport } from "../reuse-report.js";
|
|
19
20
|
import { db } from "../store/db.js";
|
|
20
21
|
import { redact } from "../redact.js";
|
|
21
22
|
import { getSyncStatus, formatSyncStatus, submitMemories, outboxDraftCount } from "../submit.js";
|
|
@@ -45,7 +46,7 @@ export const TOOL_DESCRIPTIONS = {
|
|
|
45
46
|
memory_list: "List memories in a scope, newest first, as a numbered inventory. Each line keeps the raw fields — [type] id=… created=… source=… — say them back to the user in plain words (source=user → 'you told me this'; inference → 'I inferred this, check me'; keyword → 'caught from your own wording'), never read the raw id aloud unless they ask. When the user asks 'what do you remember about me?', use scope=both and present the inventory conversationally. The user may point at an entry by its number ('delete #3'): numbers are only valid for the listing you just produced — re-run memory_list, read the candidate back in full, and get a confirmation before calling memory_forget (deletion is permanent). include=all is the audit view (superseded versions, retracted, archived also shown). `type` filters to one memory type; `limit` caps how many entries come back.",
|
|
46
47
|
memory_supersede: "Replace an existing memory with a newer version. The old memory is kept as history (status: superseded) and retrieval returns the new one. Use when a saved fact becomes outdated and should be replaced rather than duplicated. Only active memories can be superseded; the new memory inherits type, tags, and aliases unless you pass replacements. An unknown id is reported back as a failure; nothing is written.",
|
|
47
48
|
memory_forget: "Delete a memory by id. Use when the user asks to forget something. To replace an outdated fact with a corrected one instead of deleting it, use memory_supersede. Deletion is permanent: before deleting, restate the memory's content to the user and get a confirmation. With soft=true the memory is hidden instead (retracted: out of lists and search, file kept, one-way).",
|
|
48
|
-
memory_status: "Show the project memory sync pipeline: drafts waiting in the outbox (appdata), memories in the repo awaiting review or published, and any repo files not yet committed. Call this at session start, when the server reports drafts waiting for review,
|
|
49
|
+
memory_status: "Show the project memory sync pipeline: drafts waiting in the outbox (appdata), memories in the repo awaiting review or published, and any repo files not yet committed — plus the D77 reuse report: per-scope retrieved/useful rates, the most-reused memories, and \"zombies\" (retrieved but never confirmed useful). Call this at session start, when the server reports drafts waiting for review, when the user says 'sync memory' (or '同步记忆'), or when they ask what memory is pulling its weight. Then ask the user which drafts to sync. For the memories themselves use memory_list; for PR review state use memory_pr_status. (MCP server and the `open-memex sync-status` CLI; the opencode native plugin does not expose this tool.)",
|
|
49
50
|
memory_submit: "Move outbox drafts into the repo memory dir for review: copies the drafts in as proposed (or keeps a local approval), commits locally on the current branch, and moves the outbox originals out. Never creates a branch on its own — pass branch= only with the user's explicit approval for the full chain. Prints the push and PR commands — those need the user's explicit approval and are never run automatically. (MCP server and the `open-memex submit` CLI; the opencode native plugin does not expose this tool.) Unknown ids or a failed local commit abort the submit with the reason; nothing is published automatically.",
|
|
50
51
|
memory_propose: "Copy personal memories into the project outbox as review drafts. The personal originals stay put. Use this when personal knowledge should enter a project's review flow; memory_submit then moves the drafts into the repo, and memory_promote advances drafts already there. (MCP server and the `open-memex propose` CLI; the opencode native plugin does not expose this tool.)",
|
|
51
52
|
memory_promote: "Advance a project memory one step up the review ladder (proposed → approved → published), or reject it with a note. Rejected memories are never deleted — they can be revised and resubmitted. One step per call; an unknown id or a memory already published errors. Pass `by` to record the reviewer when it should differ from the default (e.g. approving on the strength of someone else's review). For brand-new outbox drafts, memory_submit is the entry point. (MCP server and the `open-memex promote` CLI; the opencode native plugin does not expose this tool.)",
|
|
@@ -493,14 +494,7 @@ export async function markMemoryUseful(args) {
|
|
|
493
494
|
};
|
|
494
495
|
}
|
|
495
496
|
const kind = args.kind ?? usefulKindFor(row.scope, row.review_state);
|
|
496
|
-
|
|
497
|
-
// recordUseful re-checks existence; both branches above agree on not-found.
|
|
498
|
-
if (!after) {
|
|
499
|
-
return {
|
|
500
|
-
title: "memory: useful not found",
|
|
501
|
-
output: `No memory with id ${args.id} in the index.`,
|
|
502
|
-
};
|
|
503
|
-
}
|
|
497
|
+
recordUseful(args.id, kind, "tool");
|
|
504
498
|
const u = getUsage(args.id);
|
|
505
499
|
const rate = reuseRate(u);
|
|
506
500
|
const useful = usefulCount(u);
|
|
@@ -514,9 +508,16 @@ export async function statusMemories() {
|
|
|
514
508
|
const st = getSyncStatus();
|
|
515
509
|
const n = st.outbox.length +
|
|
516
510
|
st.inRepo.filter((e) => e.reviewState === "proposed" || e.reviewState === "approved").length;
|
|
511
|
+
// D77-L3: the reuse report rides the status surface (the D77 text calls
|
|
512
|
+
// memory_status out by name). Personal + this project, capped for an agent
|
|
513
|
+
// result; the CLI prints the same block uncapped.
|
|
514
|
+
const reuse = loadReuseReport([
|
|
515
|
+
{ scopeKey: st.scopeKey, label: "this project" },
|
|
516
|
+
{ scopeKey: PERSONAL_SCOPE.key, label: "personal" },
|
|
517
|
+
]);
|
|
517
518
|
return {
|
|
518
519
|
title: `memory: sync status (${n} pending)`,
|
|
519
|
-
output: formatSyncStatus(st),
|
|
520
|
+
output: [formatSyncStatus(st), "", ...formatReuseReport(reuse)].join("\n"),
|
|
520
521
|
};
|
|
521
522
|
}
|
|
522
523
|
export async function submitMemoriesOp(args) {
|
package/docs/V2-DESIGN.md
CHANGED
|
@@ -580,7 +580,9 @@ requirement: personal data never touches third-party services). Benchmarks to tr
|
|
|
580
580
|
- **Phase 4 — Future, signal-gated.** Cloud `RemoteProvider` customization only on: multi-private-repo
|
|
581
581
|
sharing needs, fine-grained ACL, audit/compliance mandates · optional API-backed exporters/providers
|
|
582
582
|
for enterprise knowledge systems.
|
|
583
|
-
- **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2 ✅ built 2026-11-03 (0.9.0-alpha.2): per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool, counters in a `memory_usage` table beside the rebuildable index. D77-L3: `memory_status` reuse report
|
|
583
|
+
- **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2 ✅ built 2026-11-03 (0.9.0-alpha.2): per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool, counters in a `memory_usage` table beside the rebuildable index. D77-L3 ✅ built 2026-11-04 (0.9.0-alpha.3): `memory_status`/`sync-status` reuse report — per-scope rates, top reused, zero-reuse "zombies". D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. Cross-lingual embedding rerank (Stone 2026-10-16, from obsidian-mind comparison): ablation showed multilingual vectors beat BM25 by +12pp R@5 on the cross-lingual slice — the one gap lexical can't cover. Design as opt-in rerank for EN↔ZH queries only, not a default path; local stays BM25-first per the retrieval split above. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server. OptMem lessons (Stone 2026-10-16, from kafkaup's OptMem post): three friction-reduction ideas worth stealing from the minimalist end of the spectrum. (1) **Token cost as spec** — measure and publish open-memex's prompt/token overhead in the README, the way OptMem advertises its "426-token prompt"; acquisition copy, not just engineering. (2) **`wake` convention** — document a session-start ritual (one `memory_recall` call at the beginning of every session), mirroring OptMem's `memo wake`; the MCP tools exist but the habit isn't specified. (3) **Capture length nudge** — a soft length cap or conciseness hint at capture time, inspired by OptMem's 280-byte hard cap; forces conciseness at write time, cheap to implement, aligned with the distill-to-AGENTS.md direction (D40). Three-tier memory zones (Stone 2026-10-18, from Westenberg "I Deleted My Second Brain" + his own reflection): the second brain should have zones like cache vs database, not "keep everything" vs "delete everything". **Hot** (active zone, cache-like: small, curated, always in context — D81 aliases, WHAT-level) · **Warm** (compost/incubation: not active but not cold; periodic review, rotted items deleted — this is "forget on purpose" as engineering) · **Cold** (full archive, searchable, not resident). The hard part is the **promotion/demotion policy** — what moves between zones, when, by what rule (the cache-replacement problem for knowledge; "useful" ≠ "recently used"). Capture restraint is a separate gate: not everything deserves to enter even cold storage. Spec the policy before building the zones. Auto-capture design brief (Stone 2026-10-19, from ai-memory comparison): open-memex wants auto-capture but under the restraint philosophy (not ai-memory's capture-everything). Five design questions: (1) **End-session trigger** — what hooks does opencode actually expose? (check before designing; ai-memory built finalize-session fallback for harnesses without SessionEnd); candidates: native hook / idle timeout / explicit command / context-near-full. (2) **Distillation** — rules first (what happened: git diff, tool calls, decision sentences; zero LLM), interpretation layer (what it means) left to human/LLM. (3) **Save-value triage** — three tiers: clearly valuable → save; clearly noise → drop; ambiguous → ask human. Boundary principle: **rather over-ask than over-save** (polluted retrieval costs more than a missed memory). (4) **Routing** — four exits: project memory / personal memory / AGENTS.md proposal (distill-agents exists) / discard; keep v1 routing rules simple. (5) **Human interaction** — one batch confirmation at session end ("save these 3? [y/n/edit]"), never per-item popups. Relations: D73 keyword capture is turn-level (no conflict); `wake` convention is session-start (this is session-end — the other bookend); three-tier zones govern where auto-captured items land. Cross-agent study (Stone 2026-10-19): open-memex has cross-agent on the roadmap — study how ai-memory does it (handoff claim-once semantics, lifecycle hooks per harness, VSCode support via MCP-only) before designing ours; borrow the semantics, not the push-injection behavior (discoverability > injection). Injection cost discipline (Stone 2026-10-19, from Harness² paper arXiv:2610.14224): "ALWAYS-ON PROSE IS THE EXPENSIVE OPTION" — every always-on word competes with the agent's output budget. Current session-start injection (8 project + 5 personal) is the expensive path; direction: move "important but not needed every turn" memories to on-demand (tool/skill-triggered loading), keep only truly every-turn items resident. Tolerable now, better not to tolerate. Component cost ladder for reference: SKILL (default, zero-cost until invoked) > SCRIPT > SUBAGENT > PLUGIN ("prose can be skipped, a hook cannot") > ALWAYS-ON TEXT (last resort). Read-less compression (Stone 2026-10-19, from caveman https://github.com/JuliusBrussee/caveman): the execution layer of injection cost discipline — compress what the agent READS before it reaches context, not just what we inject. Caveman's measured numbers: 33.2% fewer input tokens per session (CSV 98.9%, logs 98.5%, YAML 99.1%, JSON 98.5% per-file; HTML has no compressor yet). Direction for open-memex: a pre-retrieval compression pass on tool outputs and retrieved passages (structured data → compact form), originals stay on disk and pullable on demand. Two design principles borrowed: (1) never compress the user's own prompt/input (Adobe CAVEWOMAN finding: cavemanning user prompts makes answers worse); (2) leave code/tool calls/payloads verbatim — compress the wrapper, not the payload. Caveats: caveman's own rules cost ~1,160 input tokens fixed; per-request billing (Copilot-style) nullifies the saving — compression pays off on per-token billing only. Knowledge upkeep as a first-class direction (Stone 2026-10-23; external validation: Hedy Zhang's FDE field retrospective on enterprise AI knowledge bases, X article 2026-10-22 — enterprise knowledge bases die on unowned upkeep, not on retrieval: stale answers served silently, conflicts with no owner, updates nobody owns after delivery). The mechanisms already exist (supersede chains, lifecycle states, `audit` health check, curator convention); the direction is driving the human cost of upkeep toward zero: (1) staleness signals — age, contradiction with newer captures, D77 zero-reuse counters — feeding a standing review queue instead of waiting for someone to remember to run `audit`; (2) a conflict inbox — semantic conflicts and broken chains as a persistent todo; (3) the endgame is D78 confidence-tier auto-curation (auto-apply the safe tier, batch-confirm the rest). Success metric: the base stays trustworthy on minutes-per-week of human attention, not a curator shift. README pitches the shipped half of this as core concept 4 (Knowledge stays current) plus a "When facts change" comparison row. Session working state note (D89, 2026-11-02) → BUILT, shipped 2026-11-03 as D90: 12th tool `memory_workingstate` on both hosts, session-start surfacing, `audit` expiry report, `wstate` CLI; opencode write path is instruction + tool (plugin API has no Stop hook).
|
|
584
|
+
|
|
585
|
+
- **Build order after D77-L3 (2026-11-04, Stone approved).** Once PR #74 (D77-L3 reuse report) merges, the queue is: (1) §20 / D88 deterministic half — `graph build` (structural edges only), compiled outline injected at session start, `graph neighbours` / `graph path` CLI, `graph lint` (orphans / stale / missing pages); no LLM, no ranking change. Fixes the live pain (knowing what is in memory without searching) and produces the lint input D78 consumes. (2) D79 feedback-weighted ranking — D77-L2 counters landed 2026-11-03, so this is newly unblocked; fixture + baseline first (measurement before mechanism). (3) D78 curate+submit — the upkeep endgame; consumes §20 lint output. (4) Embeddings gate decision (bench `--model` measures the multilingual tax) + opt-in cross-lingual rerank for EN↔ZH queries only; local stays BM25-first. (5) OptMem small items — token cost in README, `wake` session-start ritual, capture length nudge. (6) Three-tier memory zones policy spec (Hot/Warm/Cold promotion/demotion rules) before building; §20's compiled outline already covers part of Hot. (7) Auto-capture design — answer the five open questions first (notably which session-end trigger opencode exposes). Then measurement/candidates: MemoryAgentBench second external data point; Phase 2C native plugins (candidate, not committed); D9 LAN reference server likely dropped; D2/D11 iron-rule gate is a known safety gap awaiting a build/drop decision. Non-code track: Phase 2B 1–2 colleague pilot (needs people, not code). Dependency chain: §20 lint → D78; D77-L2 counters → D79; gate decision → rerank. Tension to respect: §20's resident outline must stay compact per the injection-cost discipline (2026-10-19) — "important but not every turn" stays on-demand.
|
|
584
586
|
|
|
585
587
|
- **Phase 5 — Candidates recorded 2026-10-04 (not committed).** From the competitor scan (D75): an offline FTS5 retrieval eval with a published recall number (gates any semantic-layer decision) [promoted to next build, D77] · per-client trust levels on the review ladder · cold-start Markdown import into outbox drafts · `doctor` check that each editor's MCP registration points at the intended data root (mis-rooted servers are a real failure mode elsewhere). Sequencing and scope TBD per candidate.
|
|
586
588
|
- **Phase 5 — Candidates recorded 2026-10-11 (not committed).** From the XERJ review (xerj-org/xerj, Apache-2.0; founder Ivan N requested open-memex D81 for the XERJ Corpus Hub — agreed 2026-10-11, attribution + repo link): schema-first querying (`xerj_map` — agent fetches the field map before writing query DSL; candidate: a schema/describe helper ahead of `memory_search` to cut dead queries) · token-budgeted MCP results (their `token_budget.rs` — truncate to a token budget, not a fixed row limit) · hybrid retrieval as the scale answer (BM25 + vector kNN with fused ranking, e.g. RRF — the reference pattern for the eventual embedding layer; our lexical ceiling is already measured at R@5 40.6% pooled) · NL-to-validated-query planning (`xerj_plan` — natural language in, validated query out; candidate: a query lint/validate step before search executes) · evidence-carrying memory links (typed links with evidence + confidence, soft-retire never hard-delete — same spirit as our review-state ladder; prior art for a future memory-graph direction) · honest isolation docs (state what a namespace guarantees and does NOT — query-level vs trust boundary; same spirit as our Limitations section) · cost-accounted evals (report tokens + dollars per approach, as in their case studies — the format for proving "memory makes agents cheaper"). Sequencing and scope TBD per candidate.
|
package/docs/design/decisions.md
CHANGED
|
@@ -1110,7 +1110,7 @@
|
|
|
1110
1110
|
*Status (2026-11-02): 已实现 — CHANGELOG 0.7.3 "Tool-definition quality pass (Glama TDQS 4.0/5 on v0.7.2): D76".*
|
|
1111
1111
|
|
|
1112
1112
|
- **D77 — measure usefulness by reuse, not storage (2026-10-07).** The Spark Tank submission commits the metric: "measured by reuse — fewer repeated questions, less context-prep time, approved memories reused by non-authors — not by memory count." Three layers, in build order: (1) **Offline retrieval eval.** A synthetic fixture (memories + queries with gold ids) run against the FTS5 baseline; publish recall@k/MRR in the README. This also answers the D75 "retrieval evaluation gate" open question — the number gates any future semantic-layer decision (reference points: agentmemory's published LongMemEval-S R@5 95.2%; lotargo/memory_plugin's BM25-only vs hybrid table). Later, run the MemoryAgentBench harness for directly comparable numbers (Knowl 90 / Mem0 18 / MIRIX 14 / Zep 7). (2) **Online reuse counters.** Per memory: `retrieved_count` (returned by search/list/first-turn injection) and `useful_count`, split into `useful_self` (author's own reuse — the personal-utility story) and `useful_team` (reuse of published project memories across sessions — the Spark Tank "reused by non-authors" story; no identity system needed: published state plus session diversity is the proxy). Both count as useful; reporting them separately keeps the team-knowledge claim honest. Explicit confirmation: a `memory_useful` tool call, human approve at review; behavioral: supersede/promote referencing it. Passive surfacing must never refresh usefulness — the opencode-memory lesson: unguarded exposure creates a self-reinforcing retrieval loop. Derived: `reuse_rate = useful / retrieved`; "zombie memories" (retrieved but never confirmed useful) surface in `memory_status` as decay/prune candidates. Counters live in SQLite, not frontmatter; local-only. (3) **Status surfaces.** `memory_status` gains a reuse report: top reused, zero-reuse list, per-scope reuse rate — feeding the Spark Tank pilot metrics. Moat note: competitors' "useful" is agent self-reported; our review ladder gives a human-grounded signal — approved/published means a human already said it is worth sharing. **Priority: this is the next build after 0.7.3** — fixture eval first (1–2 days, publishable number), then counters + `memory_useful`, then the status report, then the external bench.
|
|
1113
|
-
*Status (2026-11-
|
|
1113
|
+
*Status (2026-11-04): 已实现 — 三层全建。第 1 层 eval 已发;第 2 层(0.9.0-alpha.2):`memory_usage` 表(独立于可重建索引,跨 rebuild 存活)+ `retrieved_count`/`useful_self`/`useful_team` + `memory_useful` 工具与 `open-memex useful` CLI,分类代理:personal 或未评审项目草稿→self,approved/published 项目记忆→team,supersede 与评审 approve(含 `--local-approve`)各自动计一次 useful,曝光只走 search/list/injection 的显式 `record` 开关;第 3 层(0.9.0-alpha.3):`memory_status` / `sync-status` 复用报告(各作用域 retrieved/useful 比率且 self/team 分开显示、top reused、zombie=被检索却从未确认有用;superseded/deprecated 带生命周期标记,避免历史复用被读成当前指引;retracted/archived 不计入两列表,总数仍计)。仍未跑:MemoryAgentBench 第二个外部数据点(属第 1 层延伸,不在三层机制内)。*
|
|
1114
1114
|
|
|
1115
1115
|
- **D78 — pre-submit curation: human-on-the-loop, not in-the-loop (2026-10-07).** Fragmented captures must be organized before submit, but the submitter's patience is the scarce resource — every manual gate lowers the submit rate, so curation defaults to automatic and the human handles exceptions. Confidence tiers: **auto-apply** (exact duplicates, trivial fragments, inferred type/tags, title normalization); **batch-confirm** (semantic merges, splits — one summary screen, one tap to accept or undo, never per-item approval); **explicit human decision** (dropping substantive content, and submit itself — the D-hard boundary "project scope never auto-submits" stands: automation covers organizing, sharing stays human). The safety net is the supersede chain: every auto-merge preserves provenance and is undoable, so pre-approval is replaced by reversibility (undo-send, not confirm-dialog). Trigger: no new step to remember — `submit` becomes curate+submit in one command, with the curation summary attached to the PR description; session wrap-up is the alternative hook (batch-silent, per the D61 checkpoint posture). Closed loop: human undos feed back into confidence thresholds, consuming the D77 `useful_count` signals. This revises the earlier review-ladder instinct of per-item approval: the human moves from in-the-loop to on-the-loop.
|
|
1116
1116
|
*Status (2026-11-02): 未实现 — 纯设计条目:confidence 三档(auto-apply / batch-confirm / explicit human)、`submit` 变 curate+submit、undo-send 替代 pre-approval 均无代码.*
|