open-memex 0.8.0 → 0.8.2-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -0
- package/CHANGELOG.md +12 -0
- package/README.md +21 -8
- package/README.zh-CN.md +9 -2
- package/dist/.build-stamp +1 -1
- package/dist/cli.js +13 -0
- package/dist/plugin-core.js +1 -1
- package/dist/retrieve/search.js +53 -16
- package/dist/retrieve/weights.js +35 -0
- package/dist/store/db.js +42 -9
- package/dist/store/sync.js +13 -8
- package/dist/tools/ops.js +8 -2
- package/docs/V2-DESIGN.md +15 -1
- package/docs/writing-for-memory.md +94 -0
- package/package.json +1 -1
- package/scripts/smoke-mcp.ts +12 -0
- package/scripts/smoke-pure.ts +16 -0
- package/scripts/test-full.ts +49 -0
- package/server.json +2 -2
- package/src/cli.ts +15 -0
- package/src/plugin-core.ts +1 -1
- package/src/retrieve/search.ts +78 -17
- package/src/retrieve/weights.ts +37 -0
- package/src/store/db.ts +41 -9
- package/src/store/sync.ts +14 -8
- package/src/tools/ops.ts +12 -2
package/docs/V2-DESIGN.md
CHANGED
|
@@ -562,9 +562,13 @@ requirement: personal data never touches third-party services). Benchmarks to tr
|
|
|
562
562
|
- **Phase 4 — Future, signal-gated.** Cloud `RemoteProvider` customization only on: multi-private-repo
|
|
563
563
|
sharing needs, fine-grained ACL, audit/compliance mandates · optional API-backed exporters/providers
|
|
564
564
|
for enterprise knowledge systems.
|
|
565
|
-
- **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2: per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool. D77-L3: `memory_status` reuse report (top reused, zero-reuse, per-scope reuse rate) feeding the Spark Tank pilot metrics. D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server.
|
|
565
|
+
- **Near-term backlog — committed direction, not yet built (2026-10-07).** Stone's rule: anything said as "later" is written down here. D77-L2: per-memory `retrieved_count`/`useful_count` (`useful_self` vs `useful_team`) + `memory_useful` tool. D77-L3: `memory_status` reuse report (top reused, zero-reuse, per-scope reuse rate) feeding the Spark Tank pilot metrics. D77: external bench — LongMemEval-S run 2026-10-07 (`scripts/bench-longmemeval.ts`, 470 questions, recall_any@k: R@5 97.0% / MRR 0.909 per-question, R@5 40.6% on a pooled 22k corpus; ahead of agentmemory’s BM25-only, level with their BM25+vector). MemoryAgentBench still open as a second external data point. D79 follow-up SUPERSEDED → built as **D80** (2026-10-07): index-time synonym/translation expansion shipped; 41-query fixture recall@5 0.854 → 0.976, MRR 0.791 → 0.859. Embedding-model configurability (Stone 2026-10-07): the multilingual model is his personal need (EN↔ZH), not everyone’s — the model must be user-configurable. Timing: (1) NOW in the bench only — a `--model` flag to measure the multilingual tax (multilingual vs English-only on English data), which decides the DEFAULT. (2) AT SHIP TIME — `embedding.model` in `MyOMemoryConfig` + `SETTABLE_KEYS`, model-agnostic storage from day one (dim + model id stored with vectors, cache dir), download UX and `doctor` checks. Not before the gate passes: building config UX for a layer we haven’t decided to ship is premature. If multilingual tax ≈ 0, default multilingual (covers everyone); else default English-only with a docs note.D78: curate+submit — pre-submit lint, then confidence-tier auto-curation (auto-apply / batch-confirm / explicit human). D79 follow-ups: feedback-weighted ranking once D77 counters land; embeddings gate decision against the published baseline; reranker evaluation. Cross-lingual embedding rerank (Stone 2026-10-16, from obsidian-mind comparison): ablation showed multilingual vectors beat BM25 by +12pp R@5 on the cross-lingual slice — the one gap lexical can't cover. Design as opt-in rerank for EN↔ZH queries only, not a default path; local stays BM25-first per the retrieval split above. TDQS nit (separate PR): tighten search/list `type` to `z.enum`. Retrieval eval: grow the fixture (project-scope tier cases) as retrieval logic evolves. Retrieval split — local vs share server (Stone 2026-10-07): the two environments emphasize different mechanisms. Local (personal machine): BM25 + curated index-time expansion as the primary — deterministic, zero model download (D7), fast on CPU, index bloat bounded by the small corpus; embeddings opt-in only. Share server (team/org, enterprise): embeddings + ANN + reranker as the primary — server-grade compute with models resident, corpus too large and multi-author for any curated map; the curated map stays as a precision layer plus org glossary. The server additionally aggregates D77 useful_count across users into feedback-weighted ranking (already in D79 follow-ups) — a signal a single machine can never have. Bottom-up loop: curated pairs that many users converge on locally become org-glossary candidates. Economics flip: N users each downloading a model vs one model amortized on the server. OptMem lessons (Stone 2026-10-16, from kafkaup's OptMem post): three friction-reduction ideas worth stealing from the minimalist end of the spectrum. (1) **Token cost as spec** — measure and publish open-memex's prompt/token overhead in the README, the way OptMem advertises its "426-token prompt"; acquisition copy, not just engineering. (2) **`wake` convention** — document a session-start ritual (one `memory_recall` call at the beginning of every session), mirroring OptMem's `memo wake`; the MCP tools exist but the habit isn't specified. (3) **Capture length nudge** — a soft length cap or conciseness hint at capture time, inspired by OptMem's 280-byte hard cap; forces conciseness at write time, cheap to implement, aligned with the distill-to-AGENTS.md direction (D40). Three-tier memory zones (Stone 2026-10-18, from Westenberg "I Deleted My Second Brain" + his own reflection): the second brain should have zones like cache vs database, not "keep everything" vs "delete everything". **Hot** (active zone, cache-like: small, curated, always in context — D81 aliases, WHAT-level) · **Warm** (compost/incubation: not active but not cold; periodic review, rotted items deleted — this is "forget on purpose" as engineering) · **Cold** (full archive, searchable, not resident). The hard part is the **promotion/demotion policy** — what moves between zones, when, by what rule (the cache-replacement problem for knowledge; "useful" ≠ "recently used"). Capture restraint is a separate gate: not everything deserves to enter even cold storage. Spec the policy before building the zones. Auto-capture design brief (Stone 2026-10-19, from ai-memory comparison): open-memex wants auto-capture but under the restraint philosophy (not ai-memory's capture-everything). Five design questions: (1) **End-session trigger** — what hooks does opencode actually expose? (check before designing; ai-memory built finalize-session fallback for harnesses without SessionEnd); candidates: native hook / idle timeout / explicit command / context-near-full. (2) **Distillation** — rules first (what happened: git diff, tool calls, decision sentences; zero LLM), interpretation layer (what it means) left to human/LLM. (3) **Save-value triage** — three tiers: clearly valuable → save; clearly noise → drop; ambiguous → ask human. Boundary principle: **rather over-ask than over-save** (polluted retrieval costs more than a missed memory). (4) **Routing** — four exits: project memory / personal memory / AGENTS.md proposal (distill-agents exists) / discard; keep v1 routing rules simple. (5) **Human interaction** — one batch confirmation at session end ("save these 3? [y/n/edit]"), never per-item popups. Relations: D73 keyword capture is turn-level (no conflict); `wake` convention is session-start (this is session-end — the other bookend); three-tier zones govern where auto-captured items land. Cross-agent study (Stone 2026-10-19): open-memex has cross-agent on the roadmap — study how ai-memory does it (handoff claim-once semantics, lifecycle hooks per harness, VSCode support via MCP-only) before designing ours; borrow the semantics, not the push-injection behavior (discoverability > injection). Injection cost discipline (Stone 2026-10-19, from Harness² paper arXiv:2610.14224): "ALWAYS-ON PROSE IS THE EXPENSIVE OPTION" — every always-on word competes with the agent's output budget. Current session-start injection (8 project + 5 personal) is the expensive path; direction: move "important but not needed every turn" memories to on-demand (tool/skill-triggered loading), keep only truly every-turn items resident. Tolerable now, better not to tolerate. Component cost ladder for reference: SKILL (default, zero-cost until invoked) > SCRIPT > SUBAGENT > PLUGIN ("prose can be skipped, a hook cannot") > ALWAYS-ON TEXT (last resort). Read-less compression (Stone 2026-10-19, from caveman https://github.com/JuliusBrussee/caveman): the execution layer of injection cost discipline — compress what the agent READS before it reaches context, not just what we inject. Caveman's measured numbers: 33.2% fewer input tokens per session (CSV 98.9%, logs 98.5%, YAML 99.1%, JSON 98.5% per-file; HTML has no compressor yet). Direction for open-memex: a pre-retrieval compression pass on tool outputs and retrieved passages (structured data → compact form), originals stay on disk and pullable on demand. Two design principles borrowed: (1) never compress the user's own prompt/input (Adobe CAVEWOMAN finding: cavemanning user prompts makes answers worse); (2) leave code/tool calls/payloads verbatim — compress the wrapper, not the payload. Caveats: caveman's own rules cost ~1,160 input tokens fixed; per-request billing (Copilot-style) nullifies the saving — compression pays off on per-token billing only. Knowledge upkeep as a first-class direction (Stone 2026-10-23; external validation: Hedy Zhang's FDE field retrospective on enterprise AI knowledge bases, X article 2026-10-22 — enterprise knowledge bases die on unowned upkeep, not on retrieval: stale answers served silently, conflicts with no owner, updates nobody owns after delivery). The mechanisms already exist (supersede chains, lifecycle states, `audit` health check, curator convention); the direction is driving the human cost of upkeep toward zero: (1) staleness signals — age, contradiction with newer captures, D77 zero-reuse counters — feeding a standing review queue instead of waiting for someone to remember to run `audit`; (2) a conflict inbox — semantic conflicts and broken chains as a persistent todo; (3) the endgame is D78 confidence-tier auto-curation (auto-apply the safe tier, batch-confirm the rest). Success metric: the base stays trustworthy on minutes-per-week of human attention, not a curator shift. README pitches the shipped half of this as core concept 4 (Knowledge stays current) plus a "When facts change" comparison row.
|
|
566
566
|
|
|
567
567
|
- **Phase 5 — Candidates recorded 2026-10-04 (not committed).** From the competitor scan (D75): an offline FTS5 retrieval eval with a published recall number (gates any semantic-layer decision) [promoted to next build, D77] · per-client trust levels on the review ladder · cold-start Markdown import into outbox drafts · `doctor` check that each editor's MCP registration points at the intended data root (mis-rooted servers are a real failure mode elsewhere). Sequencing and scope TBD per candidate.
|
|
568
|
+
- **Phase 5 — Candidates recorded 2026-10-11 (not committed).** From the XERJ review (xerj-org/xerj, Apache-2.0; founder Ivan N requested open-memex D81 for the XERJ Corpus Hub — agreed 2026-10-11, attribution + repo link): schema-first querying (`xerj_map` — agent fetches the field map before writing query DSL; candidate: a schema/describe helper ahead of `memory_search` to cut dead queries) · token-budgeted MCP results (their `token_budget.rs` — truncate to a token budget, not a fixed row limit) · hybrid retrieval as the scale answer (BM25 + vector kNN with fused ranking, e.g. RRF — the reference pattern for the eventual embedding layer; our lexical ceiling is already measured at R@5 40.6% pooled) · NL-to-validated-query planning (`xerj_plan` — natural language in, validated query out; candidate: a query lint/validate step before search executes) · evidence-carrying memory links (typed links with evidence + confidence, soft-retire never hard-delete — same spirit as our review-state ladder; prior art for a future memory-graph direction) · honest isolation docs (state what a namespace guarantees and does NOT — query-level vs trust boundary; same spirit as our Limitations section) · cost-accounted evals (report tokens + dollars per approach, as in their case studies — the format for proving "memory makes agents cheaper"). Sequencing and scope TBD per candidate.
|
|
569
|
+
- **Phase 5 — Candidates recorded 2026-10-12 (not committed).** From the mattpocock/skills review (writing-for-agents: context pointers, two loads, information hierarchy, completion criteria): a "writing-for-memory" authoring checklist (`docs/writing-for-memory.md`) — trigger branches per entry, tldr+body layered storage, capture criteria (self-contained, dated, D2 role discipline), curation bar (real-session + D77 reuse counters), glossary as first-class memory. Sequencing and scope TBD; implement after the China trip.
|
|
570
|
+
- **Phase 5 — Candidates recorded 2026-10-24 (not committed).** From the RAW-RAG paper (arXiv:2610.30755, 22 Oct 2026; code: Litmeb/RAW-RAG): naive dense retrieval + cross-encoder reranking over raw history beats 12 agentic memory systems on 6 of 7 benchmark–model settings across LoCoMo / LongMemEval / MEME / MemoryAgentBench; ablations make reranking the hardest single lever (removing it −3.92, CI excludes zero) and show added complexity (atomic facts, summaries, graph expansion) hurting on MemoryAgentBench. Assessment: baseline claim strong, "structure is useless" claim medium, generalization to interactive agents weak (paper's own scope limit). (1) **Naive-RAG baseline in the retrieval gate** — every retrieval change must beat a well-specified RAW-RAG baseline on the same fixture before its complexity ships (measurement before mechanism, applied to baselines); MemoryAgentBench is the open second external data point from D77 — reuse its 14 sources / 2,071 questions as the second bench. (2) **Reranker gate, not default** — run BM25 + local small cross-encoder rerank on the D77 fixture and let the gate decide; local CPU + configurable model are preconditions (D7 zero-model-download for local). (3) **Raw-evidence anchors for distilled memories** — recall returns conclusion + source excerpt via canonical_ref (§3.5), so the reader can expand to original text on demand; no change to the no-raw-transcript default. (4) **Retrieve wide, then narrow** — widen the recall pool (e.g. top-50) and let lane fusion + rerank narrow to the injected 3–10, targeting the pooled-corpus gap (D77: per-question R@5 97.0% vs pooled 40.6%). (5) **Path-carrying recall for supersede chains** — when a hit is superseded, return the new conclusion plus the one-hop change reason, not the conclusion alone. Explicitly NOT adopted: storing full raw conversation history (conflicts with §3.5 no-transcript default and capture restraint); LLM-driven extraction / knowledge graphs as default write paths (require a design-doc update first, per AGENTS.md); generalizing benchmark claims to interactive agents. Sequencing and scope TBD per candidate.
|
|
571
|
+
- **Phase 5 — Candidates recorded 2026-10-25 (not committed).** From Tantivy (quickwit-oss/tantivy, Rust Lucene-style full-text library; via @realchendahuang post 2026-10-24; research pass over README, ARCHITECTURE.md, docs.rs): the comparison is Tantivy vs the current SQLite FTS5 + D80 index-time expansion + bigram cjk column, not Tantivy vs hand-rolled BM25. (1) **Chinese search-mode segmentation experiment** — index compound words plus their components (jieba cut_for_search style) against the current unigram + overlapping-bigram cjk column; bigrams are over-splitting (high recall, low precision, bloated column), word + components move toward precision while keeping partial-query recall; gate on the retrieval-eval fixture (R@5/MRR) before any default change; constraint check first: a jieba-style dependency breaks the deliberate zero-native-tokenizer design (cjk.ts runs on both sqlite runtimes). (2) **Field norms as a first-class diagnostic** — the D83 expansions-column length-norm dilution is a field-norm problem, not a weighting problem; Tantivy's per-field 1-byte log-scale fieldnorm confirms the diagnosis; no encoding change (FTS5 already normalizes per column). (3) **Columnar aggregation for the reuse report** — D77-L3 counts by scope/status/age bucket should come from a column scan, not a re-search (Tantivy's fast-field pattern). (4) **Tombstone + background reclaim as confirmed discipline** — Tantivy never mutates published segments (tombstone bitset now, merge reclaims later, manifest swap for visibility); this validates the existing soft-delete + supersede chain, and explains the wipe-path FTS5 'delete' corruption workaround as the cost of in-place mutation — no change, discipline confirmed. (5) **Fuzzy (Levenshtein 1) typo tolerance, low priority** — prefix match + synonym expansion already cover most morphological variation; agent queries rarely need typo tolerance. Already covered, not counted as borrowings: separate stores (inverted match / column filter / fetch content only for displayed hits), per-column snippet + matchedVia attribution (D83), immutable + soft-delete lifecycle. Explicitly NOT adopted: migrating to Tantivy itself (stack + dependency cost, no fixture win shown). Unverified in this pass: which modes tantivy-jieba exposes and jieba-rs dictionary size (algorithm verified from sibling jieba implementations only); Tantivy's BM25 k1/b constants (only "same as Lucene" confirmed). Sequencing and scope TBD per candidate.
|
|
568
572
|
|
|
569
573
|
## 19. Migration v1 → v2
|
|
570
574
|
|
|
@@ -1605,6 +1609,15 @@ requirement: personal data never touches third-party services). Benchmarks to tr
|
|
|
1605
1609
|
- **D81 — user-defined alias vocabulary as query-time expansion (2026-10-07).** Stone: an alias should be a *kind of memory* — defined once, remembered forever. Any memory type may carry `alias:` (the nickname, e.g. 香蕉计划) and `target:` (what it refers to, e.g. 支付系统重构项目) frontmatter; the alias memory defines vocabulary once per scope. At query time (`src/retrieve/alias-memory.ts`), query terms are looked up against the alias registry — one cheap `SELECT` per `search()` call over rows with non-empty alias+target (trigger A: exact registry hit, never a count heuristic; the D79 0/470 trigger is not repeated). On a hit the query is rewritten with the other side of the relation — bidirectional (alias→target and target→alias), depth < 3, cycle-safe — and re-run as round 2 (round 3 for chains). Rounds merge by normalized-max (`mergeRounds`): each round's BM25 scores are divided by that round's best (raw cross-round scores are incomparable — °C vs °F), then final = max, so an alias-side champion competes evenly with a literal-side champion — alias hits are never down-weighted for coming from a later round. Merged pool is re-ranked then cut to [1.5K, 2K] (Stone) — never back to K, or round-2 hits would be starved the way D79's slice-to-K starved them. No reindex when an alias changes (query-time). Scope-aware: a project codename stays in its project. *Rationale: D80's curated map is static and shared — personal vocabulary (小宝贝, 香蕉计划) doesn't belong in it; per-memory D61 aliases don't propagate to new memories. Alias memories close the gap: user-defined, scope-aware, effective immediately.* v1: user-explicit creation only (`memory_add` alias/target params, CLI `--alias`/`--target`, frontmatter edit); agent-proposed aliases deferred. D81 also rewords the init `captureAliases` prompt (it is not the synonym switch — D80's expansion always applies). Eval: 3 new fixture queries (香蕉计划↔支付系统重构项目 both directions + the alias memory itself), 44 queries total: recall@1 0.750, recall@5 **0.977** (43/44), MRR 0.857 — the single miss remains `how do we ship` (known D80 trade-off).
|
|
1606
1610
|
|
|
1607
1611
|
|
|
1612
|
+
- **D82 — closed `type` taxonomy on search/list (2026-10-07).** Follow-up deferred from the TDQS pass (#50), closes issue #51: `memory_search` / `memory_list` `type` is now `z.enum(MEMORY_TYPE_TAXONOMY)` instead of `z.string()` — a misspelled type (`"decisions"`) is rejected at the schema (MCP -32602) instead of returning zero results the agent can't distinguish from "no memories of that type". Hard rejection, not a soft "no matches" message: closed taxonomy + fail-fast. Existing rows with non-taxonomy types are possible (the frontmatter parser only coerces the v1 `instruction` → `knowledge` case) and are unaffected — they still surface in unfiltered search/list, they just can't be addressed by a type filter. The opencode plugin shares the same args objects, so both surfaces tighten together; `scripts/smoke-mcp.ts` guards the rejection. `memory_submit` keeps `destructiveHint: false` — settled in the #50 review (the outbox unlink happens after the repo copy is committed and hash-verified, so it's recoverable from git); recorded here so it isn't re-litigated.
|
|
1613
|
+
|
|
1614
|
+
- **D83 — D80 expansions get their own index column (2026-10-07).** Design follow-up to issue #56 (deferred from the D80 review): the curated index-time variants (`expansionVariantsForDoc`) move out of the `aliases` column into a dedicated `expansions` column — on `memories`, on `memories_fts`, and in the ai/ad/au triggers. `writeRow` stores user/authored aliases in `aliases` and derived variants in `expansions` (still filtered against the supplied set, still never written to Markdown); the `cjk` bigram column keeps covering body + tags + aliases + expansions so CJK variants stay substring-searchable. Retrieval: the FTS MATCH set gains the column with its own bm25 weight (at or below `aliases` — the fixture is the gate, tune don't guess); `SearchHit.aliases` and the `(aka: …)` attribution surface only user-authored aliases; `search --explain` reports which column matched (alias vs expansion). Migration: `SCHEMA_VERSION` 8 → 9; the bump triggers `rebuildIndexSchema`, which repopulates from Markdown — safe because expansions are derived and Markdown is untouched.
|
|
1615
|
+
*Update (2026-10-07, implemented and measured):* the `how do we ship` rank did **not** improve (still 8) — the dilution for expansion-column matches is inherent to per-column BM25 length norm (32 variants in their own column dilute about as much as 36 did in the shared one), not to the mixing. What the split fixed is provenance (`(aka:)` / `SearchHit.aliases` are user-authored again) and the dilution of deliberate user aliases (a ≤4-term column now scores cleanly). Regression gate passed: `retrieval-eval` R@1 0.750 / R@5 0.977 / MRR 0.857, identical to pre-split; v8→v9 migration verified on a scratch DB (merged aliases re-split correctly, `user_version` 9). *Rationale: an agent reading `(aka: …)` must be able to trust it means "alternate phrasings someone attached to this memory", and a 32-variant derived list must not dilute the weight of 4 deliberate ones.*
|
|
1616
|
+
- **D84 — `limit` is a cap on the return, never a hint (2026-10-07).** Correction to D81's cut rule, raised by the review of #52. D81 widened the merged alias pool to `[1.5K, 2K]` and passed that widened value to `resolveVisible` as the slice — so when an alias round fired, `search()` returned up to **2× `limit`** rows to every caller (`memory_search`, the MCP tool, `cli search`), with nothing in `--explain` saying so. The pool stays wider than `limit` — each round already over-fetches by `fetchLimit`, and the starvation D79's slice-to-K caused is real and still prevented — but the *return* is cut to `limit`. (D81's `1.5K` line was also dead: `max(min(pool,2K), min(pool,1.5K))` is always `min(pool,2K)`, so the documented band never existed; it is gone.) No ranking change — the merge sorts before any cut, so top-k is byte-identical; `scripts/retrieval-eval.ts` unchanged at R@1 0.750 / R@5 0.977 / MRR 0.857. *Rationale: `limit` is a contract on what a caller receives. Headroom belongs inside the ranking, not in the result set — an agent paying 2× the tool-result tokens on alias-touching queries has no way to see that it happened.*
|
|
1617
|
+
- **D86 — the `expansions` BM25 weight, set by the fixture (2026-10-07).** Closes the two D83 clauses that were written into the decision log but never landed in code (`00c4d2d` shipped `bm25(memories_fts)` with no weight vector, and left `--explain` reporting only `fts:` / `candidates:` / `alias rounds:`). (1) **Weight vector** — `bm25(memories_fts, …)` now takes an explicit `[content, tags, aliases, expansions, type, cjk]`, with `expansions` at **0.75** (`src/retrieve/weights.ts`, kept out of `search.ts` so `smoke-pure.ts` keeps its no-sqlite guarantee). The measurement that sets it: on equal-length docs at weight 1.0, a direct body match and an expansion-only match score **identically** (`-3.983` vs `-3.983`) — the preference was being decided by tiebreak, not by the ranking function. Any weight < 1.0 breaks that tie toward the body. `0.75` is the *highest* weight the checked-in fixture cannot distinguish from 1.0 (sweep: 1.0 / 0.9 / 0.75 all yield R@1 0.750 / R@5 0.977 / MRR 0.857; 0.5 already costs MRR; below 0.35 R@5 breaks), so the ordering guarantee is free and the fixture remains the gate — retune by re-running the sweep, never by intuition. `SearchHit.matchedVia` (`body` | `alias` | `expansion` | `other`) is computed from the same per-column highlights the attribution already used. (2) **`--explain` attribution** — prints `matched via: body=N alias=N expansion=N other=N` alongside the active weights, so a surprising rank is traceable to the column that caused it. *Rationale: D83 promised the split would make the two columns distinguishable; on equal-weight columns they were not distinguishable at all, so the split bought provenance without buying ranking. This is the missing half.* **Explicit non-goal:** this does *not* repair D80's `how do we ship` rank-8 case — that is per-column length norm (one variant among many in a long `expansions` field), not a weighting problem. Fixing that means narrowing *what* lands in the column, not re-weighting it. Fixture unchanged: R@1 0.750 / R@5 0.977 / MRR 0.857.
|
|
1618
|
+
- **D85 — the alias registry is indexed, not cached (2026-10-07).** From the same review (#55). D81's `loadAliasEntries` runs on **every** `search()` call over `alias <> '' AND target <> ''` — a full table scan on a corpus that may be tens of thousands. Added `idx_memories_alias ON memories(scope_key) WHERE alias <> '' AND target <> ''` to `TABLE_SCHEMA`. Two deliberate choices: (1) **no `SCHEMA_VERSION` bump** — `TABLE_SCHEMA` executes on every db open, not only on rebuild, so `CREATE INDEX IF NOT EXISTS` is idempotent and existing installs pick it up on next open with no rebuild; this also keeps the bump free for D83's 8 → 9 schema change. (2) **No result cache** — the registry must be re-read per query so a newly saved alias memory applies on the very next query (D81's freshness property). An index accelerates the lookup without caching results; those two concerns are separable and this resolves only the first. Measured at 5k memories: 971ms → 39ms per 2000 scoped lookups (~25×), `EXPLAIN QUERY PLAN` confirms `SEARCH memories USING INDEX idx_memories_alias`. Note the *unscoped* variant of the query still prefers `idx_memories_status`; `search()` always passes `scopeKeys`, so the partial index is what ships. *Rationale: the alias table is tiny by construction — one row per defined nickname per scope — so its lookup cost must be proportional to aliases defined, not to store size.*
|
|
1619
|
+
- **D87 — stale stores migrate before the schema is created (2026-10-28).** D85's "no `SCHEMA_VERSION` bump" reasoning was wrong for one population: stores created before D81 (`SCHEMA_VERSION` 7, ≤0.7.x) have no `alias`/`target` columns, and `CREATE TABLE IF NOT EXISTS` adds no columns to an existing table — so `CREATE INDEX IF NOT EXISTS idx_memories_alias ... WHERE alias <> ''` throws `no such column: alias` inside `db()` before the `userVersion < SCHEMA_VERSION` rebuild check ever runs, wedging every open (CLI, MCP, plugin) on that store. Reproduced with a faithful v7 fixture: current code throws exactly that error. Fix, ordering only: in `db()`, when the `memories` table exists but the version is old, `rebuildIndexSchema()` runs *before* `TABLE_SCHEMA`/`FTS_SCHEMA` are executed (the table exists, so rebuild's `DELETE` is valid); fresh stores still fall through to creation and the existing post-create version check is unchanged. No version bump — the guard is ordered before creation, so version 9 still covers the migration, and no new mechanism was needed. Regression test in `test-full.ts` §9 builds a v7 store and asserts the open succeeds, `user_version = 9`, and the three columns are present. The D85 comment in `db.ts` carries a correction note. *Rationale: the wipe-and-repopulate migration only works if it runs before the schema that assumes its result; the bug was ordering, not a missing mechanism.*
|
|
1620
|
+
|
|
1608
1621
|
## Open Questions
|
|
1609
1622
|
|
|
1610
1623
|
_Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped layers of the retrieval-robustness question were removed — the decisions are the record. What remains below is genuinely open._
|
|
@@ -1630,6 +1643,7 @@ _Pruned 2026-10-04 (D75): entries decided in D56/D57/D61/D68 and the shipped lay
|
|
|
1630
1643
|
|
|
1631
1644
|
- **Enterprise / pro edition: team server + remote MCP (open, 2026-10-05):** the M365 question above reframes the enterprise surface as a *team-tier* endpoint: personal memory stays local-first, team/project memory is served company-hosted and called by Copilot Studio agents as a remote MCP server. The packaging question is whether that team tier ships as an open-memex "enterprise/pro" edition (hosted service or deployable server + remote MCP) or stays a documented deployment pattern companies run themselves. Adoption insight from large-company anecdotes: internally-built programmer tools get pulled in by BAs/QAs/POs on their own — the wedge is developers, and the non-programmer entry point may not need to be built up front; it arrives via the team surface (M365/Teams) once the team server exists. Open: packaging and monetization model (Stone's current goal for open-memex is personal influence, not user count or revenue — an enterprise edition is a support burden, so timing and shape need a deliberate decision); hosting and auth model for the team server; how the personal/team split is enforced at the protocol level.
|
|
1632
1645
|
- *Refinement (2026-10-05): the entry point is not built, it is placed.* Employees already live in team chat — asking questions, getting work done. A team-chat AI agent (e.g. a Copilot Studio agent in Teams backed by the team memory server) becomes the entry point by standing where the questions already get asked. Adoption is pull, not push: the agent answers and does work in the chat, and usage follows. This dissolves the "non-programmer entry point" problem rather than solving it — there is no separate door to build.
|
|
1646
|
+
- *Interface constraint (2026-10-16, from tool/skill/MCP boundary audit):* design the MCP interface for dual use from the start — internal agents and external agents (Copilot Studio, team chat) consume the same tools. Reserve org scope alongside project/personal; the review ladder's single-reviewer assumption will need multi-reviewer semantics. No code changes now; the constraint is recorded so the team tier doesn't require an interface rework later.
|
|
1633
1647
|
|
|
1634
1648
|
---
|
|
1635
1649
|
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Writing for Memory — authoring checklist
|
|
2
|
+
|
|
3
|
+
> Status: Phase 5 candidate (not committed), recorded 2026-10-12.
|
|
4
|
+
> Adapted from mattpocock/skills `writing-for-agents`
|
|
5
|
+
> (https://github.com/mattpocock/skills).
|
|
6
|
+
> That document is a theory of writing for the agent-as-reader; this is the same
|
|
7
|
+
> theory applied to memory entries. Retrieval (FTS5/BM25, D81 alias expansion)
|
|
8
|
+
> decides *which* entries match — this checklist decides whether a matched entry
|
|
9
|
+
> *deserves* the context it costs.
|
|
10
|
+
|
|
11
|
+
## 1. Trigger branches — when this memory surfaces
|
|
12
|
+
|
|
13
|
+
A memory entry is a context pointer: its title/summary wording decides when it gets
|
|
14
|
+
retrieved, not just its body. A must-keep fact behind a weakly worded title is a
|
|
15
|
+
variance bug: sharpen the wording first.
|
|
16
|
+
|
|
17
|
+
- [ ] State what the entry is AND the branches that should trigger it (a branch is
|
|
18
|
+
a distinct situation where this memory applies — different runs take
|
|
19
|
+
different branches).
|
|
20
|
+
- [ ] Front-load the leading words: retrieval and reranking weight the front.
|
|
21
|
+
- [ ] One trigger per branch. Synonyms renaming a single branch are one branch
|
|
22
|
+
written twice — collapse them.
|
|
23
|
+
- [ ] Say when it does **not** apply. A memory with no negative scope retrieves
|
|
24
|
+
into situations it doesn't belong in.
|
|
25
|
+
|
|
26
|
+
## 2. Two budgets — what a retrieved memory costs
|
|
27
|
+
|
|
28
|
+
Every retrieved entry spends context budget on the agent's window, whether or not
|
|
29
|
+
it turns out to be useful that turn.
|
|
30
|
+
|
|
31
|
+
- [ ] Entries injected every session (high importance, always-relevant) earn the
|
|
32
|
+
hardest pruning: each line must justify permanent context spend.
|
|
33
|
+
- [ ] Entries retrieved on demand cost only the retrieval plus their own lines —
|
|
34
|
+
this is where detail belongs, not in the always-injected set.
|
|
35
|
+
- [ ] The human's curation attention is the other budget: capture is cheap,
|
|
36
|
+
curation is the price of a trustworthy store. Spend it where judgement
|
|
37
|
+
matters (D78: human-ON-the-loop).
|
|
38
|
+
|
|
39
|
+
## 3. Layered entries — tldr + body
|
|
40
|
+
|
|
41
|
+
Progressive disclosure applied to storage: split what every retrieval needs from
|
|
42
|
+
what only some retrievals need.
|
|
43
|
+
|
|
44
|
+
- [ ] Every entry carries a one-line tldr: the whole point, self-contained.
|
|
45
|
+
Retrieval would return the tldr layer first; the body would load on
|
|
46
|
+
demand. (Proposed storage design — Phase 5 candidate, not built yet.)
|
|
47
|
+
- [ ] Inline in the tldr what every branch needs; push branch-specific detail
|
|
48
|
+
into the body.
|
|
49
|
+
- [ ] Never bury the action: if the entry exists to change behavior, the tldr
|
|
50
|
+
says what to do, the body says why.
|
|
51
|
+
|
|
52
|
+
## 4. Capture criteria — is the entry self-contained?
|
|
53
|
+
|
|
54
|
+
A memory that needs its original session to be understood is a pointer to
|
|
55
|
+
nothing. Each entry must answer, on its own:
|
|
56
|
+
|
|
57
|
+
- [ ] What happened / what was decided (the fact, not the story).
|
|
58
|
+
- [ ] When it applies (trigger branches, §1).
|
|
59
|
+
- [ ] When it does NOT apply (negative scope, §1).
|
|
60
|
+
- [ ] Source: which session/decision it came from, dated. Undated lore rots;
|
|
61
|
+
the reader can't tell stale from live.
|
|
62
|
+
- [ ] Role discipline (D2): records are records. An entry that reads as an
|
|
63
|
+
instruction must carry `role: instruction` explicitly — never let a history
|
|
64
|
+
entry impersonate a current rule.
|
|
65
|
+
|
|
66
|
+
## 5. Co-location — one entry, one meaning
|
|
67
|
+
|
|
68
|
+
- [ ] A memory's context, decision, and caveats live under one entry, not
|
|
69
|
+
scattered across three. Reading one part should bring its neighbours.
|
|
70
|
+
- [ ] Distinct from duplication: repeating one meaning in two entries is a merge
|
|
71
|
+
candidate; fragmenting one meaning across entries is the failure mode.
|
|
72
|
+
- [ ] Prefer editing the existing entry over appending a contradicting one.
|
|
73
|
+
Contradictions without resolution are retrieval poison — resolve,
|
|
74
|
+
supersede, or deprecate (`status: deprecated`); never hard-delete.
|
|
75
|
+
|
|
76
|
+
## 6. Curation bar — what earns a place in long-term memory
|
|
77
|
+
|
|
78
|
+
Capture-first means the inbox is generous; the long-term store is not. An entry
|
|
79
|
+
stays only if it clears both:
|
|
80
|
+
|
|
81
|
+
1. **Recorded from a real session.** What happened, what was decided, what went
|
|
82
|
+
wrong — with a date and source. A hypothetical ("it would be nice to
|
|
83
|
+
remember…") does not clear this bar.
|
|
84
|
+
2. **Retrieval earns its keep.** D77 counters (`retrieved_count`/`useful_count`,
|
|
85
|
+
not yet built — near-term backlog) are the instrument: entries with sustained
|
|
86
|
+
zero-reuse are prune/merge candidates. Attention, not sentiment, decides.
|
|
87
|
+
|
|
88
|
+
## 7. Glossary discipline — shared vocabulary as first-class memory
|
|
89
|
+
|
|
90
|
+
- [ ] Project glossaries (D81 alias vocabulary) are curated memory, not just
|
|
91
|
+
query-time expansion machinery: terms, what they mean here, and what they
|
|
92
|
+
are NOT.
|
|
93
|
+
- [ ] New entries should use the glossary's terms. A new term coined in an entry
|
|
94
|
+
without a glossary home is an alias-expansion miss waiting to happen.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "open-memex",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.2-alpha.2",
|
|
4
4
|
"mcpName": "io.github.stoneskin/open-memex",
|
|
5
5
|
"description": "Local-first memory layer and protocol for AI coding agents. Markdown source of truth, SQLite FTS5 index, zero cloud.",
|
|
6
6
|
"keywords": [
|
package/scripts/smoke-mcp.ts
CHANGED
|
@@ -133,6 +133,18 @@ try {
|
|
|
133
133
|
const listBoth = await req("tools/call", { name: "memory_list", arguments: { scope: "both" } });
|
|
134
134
|
const listBothText = listBoth.result?.content?.[0]?.text ?? "";
|
|
135
135
|
check("memory_list both scopes (D68)", listBothText.includes("About you"));
|
|
136
|
+
// D82: closed type taxonomy — a misspelled type is rejected at the schema,
|
|
137
|
+
// never silently empty.
|
|
138
|
+
const badType = await req("tools/call", {
|
|
139
|
+
name: "memory_search",
|
|
140
|
+
arguments: { query: "mcp e2e probe", type: "decisions" },
|
|
141
|
+
});
|
|
142
|
+
const badTypeText: string = badType.result?.content?.[0]?.text ?? "";
|
|
143
|
+
check(
|
|
144
|
+
"memory_search rejects misspelled type (D82)",
|
|
145
|
+
/-32602/.test(badTypeText) && /Invalid option/.test(badTypeText),
|
|
146
|
+
badTypeText.slice(0, 160),
|
|
147
|
+
);
|
|
136
148
|
// D70: numbers run across the whole listing, so "delete #3" names one memory.
|
|
137
149
|
// Needs a personal entry too, or there is only one section to number.
|
|
138
150
|
const personalAdd = await req("tools/call", {
|
package/scripts/smoke-pure.ts
CHANGED
|
@@ -9,6 +9,7 @@ import { resolveProjectScope, resolveCwdScope, pickScopeRoot, PERSONAL_SCOPE } f
|
|
|
9
9
|
import { cjkIndexText, cjkQueryExpr, hasCjk } from "../src/retrieve/cjk.ts";
|
|
10
10
|
import { toFtsQuery } from "../src/retrieve/query.ts";
|
|
11
11
|
import { expansionVariantsForDoc, MAX_AUTO_VARIANTS } from "../src/retrieve/synonyms.ts";
|
|
12
|
+
import { BM25_WEIGHTS, DEFAULT_EXPANSIONS_WEIGHT } from "../src/retrieve/weights.ts";
|
|
12
13
|
import {
|
|
13
14
|
planAliasExpansions,
|
|
14
15
|
rewriteQueryOnce,
|
|
@@ -949,5 +950,20 @@ console.log("== D81 alias-memory query-time expansion ==");
|
|
|
949
950
|
ok("single round orders by score", single[0].doc.id === "b" && single[0].score === 1, JSON.stringify(single.map((m) => [m.doc.id, m.score])));
|
|
950
951
|
}
|
|
951
952
|
|
|
953
|
+
console.log("== D83 expansions BM25 weight + matchedVia ==");
|
|
954
|
+
{
|
|
955
|
+
// The weight vector must stay in memories_fts declaration order:
|
|
956
|
+
// [content, tags, aliases, expansions, type, cjk]. A mis-ordered vector
|
|
957
|
+
// silently down-weights the wrong column, so pin the shape.
|
|
958
|
+
ok("BM25_WEIGHTS has one entry per fts column", BM25_WEIGHTS.length === 6, JSON.stringify(BM25_WEIGHTS));
|
|
959
|
+
ok("expansions is index 3", BM25_WEIGHTS[3] === DEFAULT_EXPANSIONS_WEIGHT, JSON.stringify(BM25_WEIGHTS));
|
|
960
|
+
// D83: expansions sits strictly below aliases, so a body match outranks an
|
|
961
|
+
// expansion-only match by construction instead of by tiebreak.
|
|
962
|
+
ok("expansions weighted below aliases", BM25_WEIGHTS[3] < BM25_WEIGHTS[2], `${BM25_WEIGHTS[3]} vs ${BM25_WEIGHTS[2]}`);
|
|
963
|
+
ok("expansions keeps non-zero weight (D80 rescue slice must survive)",
|
|
964
|
+
BM25_WEIGHTS[3] > 0, String(BM25_WEIGHTS[3]));
|
|
965
|
+
ok("body stays the strongest column", BM25_WEIGHTS[0] >= BM25_WEIGHTS[3], JSON.stringify(BM25_WEIGHTS));
|
|
966
|
+
}
|
|
967
|
+
|
|
952
968
|
console.log(fails === 0 ? "\nALL PASS" : `\n${fails} FAILURES`);
|
|
953
969
|
process.exit(fails === 0 ? 0 : 1);
|
package/scripts/test-full.ts
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
// parsed, and crash-prone commands retry up to 5 times.
|
|
10
10
|
import { execFileSync, spawn } from "node:child_process";
|
|
11
11
|
import fs from "node:fs";
|
|
12
|
+
import { createRequire } from "node:module";
|
|
12
13
|
import os from "node:os";
|
|
13
14
|
import path from "node:path";
|
|
14
15
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
@@ -414,6 +415,54 @@ ok("migrate personal→project runs", r.code === 0, (r.err || r.out).slice(0, 15
|
|
|
414
415
|
walk(memRoot);
|
|
415
416
|
ok("migrated file is now scope: project", /^scope: project$/m.test(moved), moved.slice(0, 200));
|
|
416
417
|
}
|
|
418
|
+
// D87: a pre-D81 index.db (SCHEMA_VERSION 7 — no alias/target/expansions
|
|
419
|
+
// columns) must open and migrate instead of wedging every command with
|
|
420
|
+
// "no such column: alias": D85's partial index throws before the version
|
|
421
|
+
// check runs. Build a faithful v7 store, then any command that opens the
|
|
422
|
+
// DB must succeed and leave the schema at version 9.
|
|
423
|
+
{
|
|
424
|
+
const require = createRequire(import.meta.url);
|
|
425
|
+
const Database = require("better-sqlite3");
|
|
426
|
+
fs.rmSync(dbPath, { force: true });
|
|
427
|
+
fs.rmSync(dbPath + "-wal", { force: true });
|
|
428
|
+
fs.rmSync(dbPath + "-shm", { force: true });
|
|
429
|
+
const old = new Database(dbPath);
|
|
430
|
+
old.exec(`CREATE TABLE memories (
|
|
431
|
+
id TEXT PRIMARY KEY,
|
|
432
|
+
scope_key TEXT NOT NULL,
|
|
433
|
+
scope TEXT NOT NULL,
|
|
434
|
+
visibility TEXT NOT NULL DEFAULT 'private',
|
|
435
|
+
project_name TEXT NOT NULL DEFAULT '',
|
|
436
|
+
type TEXT NOT NULL DEFAULT 'fact',
|
|
437
|
+
role TEXT NOT NULL DEFAULT 'knowledge',
|
|
438
|
+
importance TEXT NOT NULL DEFAULT 'normal',
|
|
439
|
+
status TEXT NOT NULL DEFAULT 'active',
|
|
440
|
+
tags TEXT NOT NULL DEFAULT '',
|
|
441
|
+
aliases TEXT NOT NULL DEFAULT '',
|
|
442
|
+
content TEXT NOT NULL,
|
|
443
|
+
cjk TEXT NOT NULL DEFAULT '',
|
|
444
|
+
content_hash TEXT NOT NULL DEFAULT '',
|
|
445
|
+
superseded_by TEXT,
|
|
446
|
+
source TEXT NOT NULL DEFAULT '',
|
|
447
|
+
file_path TEXT NOT NULL,
|
|
448
|
+
mtime_ms REAL NOT NULL,
|
|
449
|
+
created_at INTEGER NOT NULL,
|
|
450
|
+
updated_at INTEGER NOT NULL,
|
|
451
|
+
review_state TEXT NOT NULL DEFAULT 'draft'
|
|
452
|
+
);`);
|
|
453
|
+
old.exec("PRAGMA user_version = 7");
|
|
454
|
+
old.prepare("INSERT INTO memories (id, scope_key, scope, file_path, mtime_ms, created_at, updated_at, content) VALUES (?,?,?,?,?,?,?,?)")
|
|
455
|
+
.run("01D87OLD", "personal", "personal", path.join(T, "gone.md"), 1, 1, 1, "stale v7 row");
|
|
456
|
+
old.close();
|
|
457
|
+
r = cliRetry(["list", "--scope", "personal"], PROJ);
|
|
458
|
+
ok("D87: pre-D81 index.db opens instead of wedging", r.code === 0 && !/no such column/.test(r.out + r.err), (r.err || r.out).slice(0, 200));
|
|
459
|
+
const after = new Database(dbPath, { readonly: true });
|
|
460
|
+
const ver = (after.prepare("PRAGMA user_version").get() as { user_version: number }).user_version;
|
|
461
|
+
const cols = (after.prepare("PRAGMA table_info(memories)").all() as { name: string }[]).map((c) => c.name);
|
|
462
|
+
after.close();
|
|
463
|
+
ok("D87: stale store migrates to schema v9", ver === 9, `user_version=${ver}`);
|
|
464
|
+
ok("D87: migrated table has alias/target/expansions", ["alias", "target", "expansions"].every((c) => cols.includes(c)), cols.join(","));
|
|
465
|
+
}
|
|
417
466
|
|
|
418
467
|
// ---------- 10. capture / doctor / mcp --print-config / init ----------
|
|
419
468
|
console.log("== capture / doctor / mcp / init ==");
|
package/server.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"name": "io.github.stoneskin/open-memex",
|
|
4
4
|
"title": "open-memex",
|
|
5
5
|
"description": "Local-first memory for AI coding agents — Markdown + SQLite FTS5, shared across editors via MCP.",
|
|
6
|
-
"version": "0.8.
|
|
6
|
+
"version": "0.8.1",
|
|
7
7
|
"repository": {
|
|
8
8
|
"url": "https://github.com/stoneskin/open-memex",
|
|
9
9
|
"source": "github"
|
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
{
|
|
14
14
|
"registryType": "npm",
|
|
15
15
|
"identifier": "open-memex",
|
|
16
|
-
"version": "0.8.
|
|
16
|
+
"version": "0.8.1",
|
|
17
17
|
"runtimeHint": "npx",
|
|
18
18
|
"packageArguments": [
|
|
19
19
|
{
|
package/src/cli.ts
CHANGED
|
@@ -10,6 +10,7 @@ import { proposeMemories, promoteMemory, listConflicts, resolveConflict, formatR
|
|
|
10
10
|
import { getSyncStatus, formatSyncStatus, submitMemories } from "./submit.ts";
|
|
11
11
|
import { getPrStatus, formatPrStatus, applyPrStatus } from "./github.ts";
|
|
12
12
|
import { search, list, countList, hitStateLabel } from "./retrieve/search.ts";
|
|
13
|
+
import { DEFAULT_EXPANSIONS_WEIGHT } from "./retrieve/weights.ts";
|
|
13
14
|
import { formatInventoryLine, truncationNote } from "./retrieve/display.ts";
|
|
14
15
|
import {
|
|
15
16
|
writeMemoryFile,
|
|
@@ -1028,6 +1029,20 @@ async function main() {
|
|
|
1028
1029
|
if (stats.aliasExpansions.length > 0) {
|
|
1029
1030
|
console.log(`alias rounds: ${stats.aliasExpansions.map((q) => `"${q}"`).join(" → ")}`);
|
|
1030
1031
|
}
|
|
1032
|
+
// D83: which indexed column each hit matched on. `expansion` means the
|
|
1033
|
+
// hit only surfaced through a D80 derived variant, so it ranks below an
|
|
1034
|
+
// equal body match by construction (BM25_WEIGHTS) — this is how a
|
|
1035
|
+
// surprising rank gets traced instead of guessed at.
|
|
1036
|
+
if (hits.length > 0) {
|
|
1037
|
+
const via = hits.reduce<Record<string, number>>((acc, h) => {
|
|
1038
|
+
acc[h.matchedVia] = (acc[h.matchedVia] ?? 0) + 1;
|
|
1039
|
+
return acc;
|
|
1040
|
+
}, {});
|
|
1041
|
+
console.log(
|
|
1042
|
+
`matched via: ${Object.entries(via).map(([k, n]) => `${k}=${n}`).join(", ")}` +
|
|
1043
|
+
` (bm25 weights: content=1 tags=1 aliases=1 expansions=${DEFAULT_EXPANSIONS_WEIGHT} type=1 cjk=1)`,
|
|
1044
|
+
);
|
|
1045
|
+
}
|
|
1031
1046
|
}
|
|
1032
1047
|
if (hits.length === 0) {
|
|
1033
1048
|
console.log("(no matches)");
|
package/src/plugin-core.ts
CHANGED
|
@@ -66,7 +66,7 @@ export function bootstrapPlugin(root: string): PluginState {
|
|
|
66
66
|
console.warn(
|
|
67
67
|
`[open-memex] found memories under legacy scope key "${cwdScope.key}". ` +
|
|
68
68
|
`Current project scope is "${scope.key}". ` +
|
|
69
|
-
`To move them: npm run cli -- migrate --from ${cwdScope.key}`,
|
|
69
|
+
`To move them: npm run cli -- migrate --from ${cwdScope.key} --to ${scope.key}`,
|
|
70
70
|
);
|
|
71
71
|
}
|
|
72
72
|
} catch (err) {
|
package/src/retrieve/search.ts
CHANGED
|
@@ -1,19 +1,28 @@
|
|
|
1
1
|
import { db } from "../store/db.ts";
|
|
2
2
|
import { toFtsQuery } from "./query.ts";
|
|
3
|
+
import { BM25_WEIGHTS } from "./weights.ts";
|
|
3
4
|
import {
|
|
4
5
|
loadAliasEntries,
|
|
5
6
|
planAliasExpansions,
|
|
6
7
|
mergeRounds,
|
|
7
8
|
} from "./alias-memory.ts";
|
|
8
9
|
|
|
10
|
+
/** D83: which indexed column a hit's match came from. */
|
|
11
|
+
export type MatchedVia = "body" | "alias" | "expansion" | "other";
|
|
12
|
+
|
|
9
13
|
export interface SearchHit {
|
|
10
14
|
id: string;
|
|
11
15
|
scope_key: string;
|
|
12
16
|
project_name: string;
|
|
13
17
|
type: string;
|
|
14
18
|
tags: string[];
|
|
15
|
-
/** D61: capture-time aliases stored on this memory (search hits only).
|
|
19
|
+
/** D61: capture-time aliases stored on this memory (search hits only).
|
|
20
|
+
* D83: user/authored only — D80 derived variants live in `expansions`. */
|
|
16
21
|
aliases: string[];
|
|
22
|
+
/** D83: which FTS column produced this hit's match — `body`, `alias`,
|
|
23
|
+
* `expansion`, or `other` (tags/type/cjk). Surfaces in `search --explain`
|
|
24
|
+
* so a surprising rank can be traced to the column that caused it. */
|
|
25
|
+
matchedVia: MatchedVia;
|
|
17
26
|
snippet: string;
|
|
18
27
|
score: number;
|
|
19
28
|
updated_at: number;
|
|
@@ -31,6 +40,11 @@ interface RawRow {
|
|
|
31
40
|
type: string;
|
|
32
41
|
tags: string;
|
|
33
42
|
aliases?: string;
|
|
43
|
+
/** D83: D80 index-time synonym/translation variants — derived, never user-authored. */
|
|
44
|
+
expansions?: string;
|
|
45
|
+
/** D83: per-column FTS highlight, so attribution knows which column matched. */
|
|
46
|
+
alias_hl?: string;
|
|
47
|
+
exp_hl?: string;
|
|
34
48
|
updated_at: number;
|
|
35
49
|
snippet: string;
|
|
36
50
|
score: number;
|
|
@@ -44,10 +58,34 @@ interface RawRow {
|
|
|
44
58
|
function toHit(r: RawRow): SearchHit {
|
|
45
59
|
const aliases = r.aliases ? r.aliases.split(",").map((a) => a.trim()).filter(Boolean) : [];
|
|
46
60
|
let snippet = r.snippet ?? "";
|
|
47
|
-
//
|
|
48
|
-
//
|
|
49
|
-
// hit otherwise looks unrelated to
|
|
50
|
-
|
|
61
|
+
// D83: attribution is per-column now. The aliases column holds only
|
|
62
|
+
// user/authored phrasings (D61); the expansions column holds D80 derived
|
|
63
|
+
// variants. Say which one matched — the hit otherwise looks unrelated to
|
|
64
|
+
// the query when the body shows no highlight.
|
|
65
|
+
const hlTerms = (s: string | undefined): string[] => {
|
|
66
|
+
if (!s) return [];
|
|
67
|
+
const m = s.match(/\[([^\]]+)\]/g);
|
|
68
|
+
return m ? [...new Set(m.map((x) => x.slice(1, -1)))] : [];
|
|
69
|
+
};
|
|
70
|
+
const aliasTerms = hlTerms(r.alias_hl);
|
|
71
|
+
const expTerms = hlTerms(r.exp_hl);
|
|
72
|
+
// D83: attribute the hit to the column that matched, so `--explain` can
|
|
73
|
+
// trace an unexpected rank. Order matters: a body highlight is the most
|
|
74
|
+
// direct evidence, then an authored alias, then a derived expansion.
|
|
75
|
+
const matchedVia: MatchedVia = /\[[^\]]+\]/.test(snippet)
|
|
76
|
+
? "body"
|
|
77
|
+
: aliasTerms.length > 0
|
|
78
|
+
? "alias"
|
|
79
|
+
: expTerms.length > 0
|
|
80
|
+
? "expansion"
|
|
81
|
+
: "other";
|
|
82
|
+
if (aliasTerms.length > 0 && aliases.length > 0) {
|
|
83
|
+
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
84
|
+
} else if (expTerms.length > 0) {
|
|
85
|
+
snippet += ` (synonym: ${expTerms.join(", ")})`;
|
|
86
|
+
} else if (aliases.length > 0 && !/\[[^\]]+\]/.test(snippet)) {
|
|
87
|
+
// Match came through another column (tags/type/cjk) — the authored
|
|
88
|
+
// aliases are still honest context for the hit.
|
|
51
89
|
snippet += ` (aka: ${aliases.join(", ")})`;
|
|
52
90
|
}
|
|
53
91
|
return {
|
|
@@ -57,6 +95,7 @@ function toHit(r: RawRow): SearchHit {
|
|
|
57
95
|
type: r.type,
|
|
58
96
|
tags: r.tags ? r.tags.split(",").filter(Boolean) : [],
|
|
59
97
|
aliases,
|
|
98
|
+
matchedVia,
|
|
60
99
|
snippet,
|
|
61
100
|
score: r.score,
|
|
62
101
|
updated_at: r.updated_at,
|
|
@@ -87,6 +126,9 @@ export function hitStateLabel(h: Pick<SearchHit, "scope_key" | "review_state">):
|
|
|
87
126
|
return ` [${h.review_state || "draft"}]`;
|
|
88
127
|
}
|
|
89
128
|
|
|
129
|
+
// D83: BM25_WEIGHTS / DEFAULT_EXPANSIONS_WEIGHT live in ./weights.ts, not
|
|
130
|
+
// here — smoke-pure.ts guards them and must stay free of any sqlite import.
|
|
131
|
+
|
|
90
132
|
/** Optional diagnostics out-param for `search --explain` (D57). */
|
|
91
133
|
export interface SearchStats {
|
|
92
134
|
ftsQuery: string;
|
|
@@ -174,6 +216,12 @@ export function search(
|
|
|
174
216
|
stopwords?: boolean;
|
|
175
217
|
orJoin?: boolean;
|
|
176
218
|
prefix?: boolean;
|
|
219
|
+
/**
|
|
220
|
+
* D83: override the BM25 weight of the `expansions` column. Omitted =
|
|
221
|
+
* `DEFAULT_EXPANSIONS_WEIGHT`. An ablation knob — it exists so the
|
|
222
|
+
* fixture can pick the value; not a per-call tuning surface.
|
|
223
|
+
*/
|
|
224
|
+
expansionsWeight?: number;
|
|
177
225
|
} = {},
|
|
178
226
|
): SearchHit[] {
|
|
179
227
|
const qflags = { stopwords: opts.stopwords, orJoin: opts.orJoin, prefix: opts.prefix };
|
|
@@ -198,10 +246,12 @@ export function search(
|
|
|
198
246
|
const typeFilter = opts.type ? ` AND m.type = ?` : "";
|
|
199
247
|
|
|
200
248
|
const sql = `
|
|
201
|
-
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.updated_at,
|
|
249
|
+
SELECT m.id, m.scope_key, m.project_name, m.type, m.tags, m.aliases, m.expansions, m.updated_at,
|
|
202
250
|
m.status, m.superseded_by, m.review_state, m.source, m.created_at,
|
|
203
251
|
snippet(memories_fts, 0, '[', ']', ' ... ', 12) AS snippet,
|
|
204
|
-
|
|
252
|
+
highlight(memories_fts, 2, '[', ']') AS alias_hl,
|
|
253
|
+
highlight(memories_fts, 3, '[', ']') AS exp_hl,
|
|
254
|
+
bm25(memories_fts, ?, ?, ?, ?, ?, ?) AS score
|
|
205
255
|
FROM memories_fts
|
|
206
256
|
JOIN memories m ON m.rowid = memories_fts.rowid
|
|
207
257
|
WHERE memories_fts MATCH ?${scopeFilter}${typeFilter}
|
|
@@ -214,6 +264,10 @@ export function search(
|
|
|
214
264
|
project_name: string;
|
|
215
265
|
type: string;
|
|
216
266
|
tags: string;
|
|
267
|
+
aliases?: string;
|
|
268
|
+
expansions?: string;
|
|
269
|
+
alias_hl?: string;
|
|
270
|
+
exp_hl?: string;
|
|
217
271
|
updated_at: number;
|
|
218
272
|
status: string;
|
|
219
273
|
superseded_by: string | null;
|
|
@@ -223,8 +277,15 @@ export function search(
|
|
|
223
277
|
snippet: string;
|
|
224
278
|
score: number;
|
|
225
279
|
};
|
|
280
|
+
// D83: the weight vector is bound per statement, so it leads the parameter
|
|
281
|
+
// list (it appears in the SELECT list, before the WHERE's MATCH).
|
|
282
|
+
const weights =
|
|
283
|
+
opts.expansionsWeight === undefined
|
|
284
|
+
? BM25_WEIGHTS
|
|
285
|
+
: BM25_WEIGHTS.map((w, i) => (i === 3 ? opts.expansionsWeight! : w));
|
|
226
286
|
const fetchFts = (matchExpr: string): RawRow[] => {
|
|
227
|
-
const params: unknown[] = [
|
|
287
|
+
const params: unknown[] = [...weights];
|
|
288
|
+
params.push(matchExpr);
|
|
228
289
|
if (opts.scopeKeys && opts.scopeKeys.length > 0) params.push(...opts.scopeKeys);
|
|
229
290
|
if (opts.type) params.push(opts.type);
|
|
230
291
|
params.push(fetchLimit);
|
|
@@ -244,7 +305,6 @@ export function search(
|
|
|
244
305
|
const aliasEntries = loadAliasEntries(opts.scopeKeys);
|
|
245
306
|
const expansions = planAliasExpansions(query, aliasEntries);
|
|
246
307
|
let merged: RawRow[] = raw;
|
|
247
|
-
let finalLimit = limit;
|
|
248
308
|
if (expansions.length > 0) {
|
|
249
309
|
const rounds: RawRow[][] = [raw];
|
|
250
310
|
for (const eq of expansions) {
|
|
@@ -255,20 +315,21 @@ export function search(
|
|
|
255
315
|
// divided by that round's best, so an alias-side champion competes
|
|
256
316
|
// evenly with a literal-side champion — no down-weighting for being
|
|
257
317
|
// from a later round.
|
|
318
|
+
//
|
|
319
|
+
// The merged pool is deliberately wider than `limit` (each round already
|
|
320
|
+
// over-fetches by fetchLimit) so round-2 hits are never starved the way
|
|
321
|
+
// D79's slice-to-K starved them. #53: that headroom is for RANKING only —
|
|
322
|
+
// the return is still cut to the caller's `limit`, since `limit` is a cap
|
|
323
|
+
// on what every caller (tools, MCP, cli) receives, not a hint.
|
|
258
324
|
merged = mergeRounds(rounds).map((m) => ({ ...m.doc, score: m.score }));
|
|
259
|
-
// Cut to [1.5K, 2K] — never back to K, or round-2 hits would be
|
|
260
|
-
// starved the way D79's slice-to-K starved them.
|
|
261
|
-
const pool = merged.length;
|
|
262
|
-
finalLimit = Math.min(pool, 2 * limit);
|
|
263
|
-
finalLimit = Math.max(finalLimit, Math.min(pool, Math.ceil(1.5 * limit)));
|
|
264
325
|
if (opts.stats) opts.stats.aliasExpansions = expansions;
|
|
265
326
|
}
|
|
266
327
|
|
|
267
328
|
// D80: the curated static map is solved at index time (synonyms.ts
|
|
268
|
-
// expands variants into the aliases column at sync time), so
|
|
269
|
-
// user alias the query side stays a single FTS round.
|
|
329
|
+
// expands variants into the indexed aliases column at sync time), so
|
|
330
|
+
// without a user alias the query side stays a single FTS round.
|
|
270
331
|
if (opts.stats) opts.stats.candidates = merged.length;
|
|
271
|
-
return resolveVisible(merged,
|
|
332
|
+
return resolveVisible(merged, limit, opts.stats);
|
|
272
333
|
}
|
|
273
334
|
|
|
274
335
|
export function list(
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* D83: BM25 column weights for the FTS index.
|
|
3
|
+
*
|
|
4
|
+
* Split out of `search.ts` deliberately: `smoke-pure.ts` must keep running
|
|
5
|
+
* with no SQLite dependency (its header says "pure-logic, no
|
|
6
|
+
* bun:sqlite"), and importing `search.ts` pulls in `store/db.ts` at module
|
|
7
|
+
* load. This module holds the tunable constant and nothing else, so the
|
|
8
|
+
* guard test can import it without opening a database.
|
|
9
|
+
*
|
|
10
|
+
* `expansions` (D80-derived synonym/translation variants) sits strictly below
|
|
11
|
+
* `aliases` so the preference is explicit rather than incidental: a body
|
|
12
|
+
* match outranks an expansion-only match, and a user-authored alias outranks
|
|
13
|
+
* a derived one. This is not cosmetic — measured on equal-length docs, at
|
|
14
|
+
* weight 1.0 a direct body match and an expansion-only match score
|
|
15
|
+
* *identically* (-3.983 vs -3.983), leaving the order to tiebreaks. Any
|
|
16
|
+
* weight < 1.0 breaks that tie in the right direction.
|
|
17
|
+
*
|
|
18
|
+
* The value is the highest one the checked-in fixture cannot distinguish from
|
|
19
|
+
* 1.0 (`scripts/retrieval-eval.ts`: R@1 0.750 / R@5 0.977 / MRR 0.857 hold
|
|
20
|
+
* from 1.0 down to 0.75; 0.5 already costs MRR, and below 0.35 R@5 breaks).
|
|
21
|
+
* So the ordering guarantee is free, and the fixture is the gate that keeps
|
|
22
|
+
* it honest — retune by re-running the sweep, not by feel.
|
|
23
|
+
*
|
|
24
|
+
* Known non-fix: this does NOT repair D80's `how do we ship` rank-8 case.
|
|
25
|
+
* That is per-column length norm (one variant among many in a long
|
|
26
|
+
* `expansions` field), not a weighting problem; see D83's update.
|
|
27
|
+
*/
|
|
28
|
+
|
|
29
|
+
/** Weight for the D80 `expansions` column. */
|
|
30
|
+
export const DEFAULT_EXPANSIONS_WEIGHT = 0.75;
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Full weight vector for `memories_fts`, in declaration order.
|
|
34
|
+
* MUST stay in sync with FTS_SCHEMA column order in `store/db.ts`:
|
|
35
|
+
* `[content, tags, aliases, expansions, type, cjk]`.
|
|
36
|
+
*/
|
|
37
|
+
export const BM25_WEIGHTS: readonly number[] = [1, 1, 1, DEFAULT_EXPANSIONS_WEIGHT, 1, 1];
|