clawmem 0.21.0 → 0.23.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -105,9 +105,9 @@ All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculat
105
105
  | `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why". |
106
106
  | `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
107
107
  | `profile` | User profile (static facts + dynamic context). |
108
- | `memory_pin` | +0.3 composite boost. PROACTIVELY for constraints, architecture decisions, corrections. |
108
+ | `memory_pin` | Lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on composite surfaces; exact-tie precedence on raw vector routes). PROACTIVELY for constraints, architecture decisions, corrections. |
109
109
  | `memory_snooze` | PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
110
- | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
110
+ | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. Weak matches return a disambiguation list instead of acting (v0.23.0). |
111
111
  | `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
112
112
  | `timeline` | Temporal neighborhood around a doc. Progressive disclosure: search → timeline → get. |
113
113
  | `memory_evolution_status` | How a doc's A-MEM metadata evolved over time. |
@@ -135,14 +135,14 @@ The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector v
135
135
 
136
136
  ## Composite scoring
137
137
 
138
- Applied automatically to all search results.
138
+ Applied on the composite surfaces: hooks, `query`, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. **v0.22.0: MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine instead** (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default); recency-intent queries keep composite everywhere. **v0.23.0:** `searchScore` on FTS surfaces is the monotonic `|bm25|/(1+|bm25|)` transform (it was a constant 1.0 through v0.22.0 due to a clamp bug — keyword relevance contributed zero ordering); FTS-transform scores and cosines are independent monotonic signals, not one calibrated scale.
139
139
 
140
140
  ```
141
141
  compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
142
142
  ```
143
143
 
144
- - `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× … 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive**.
145
- - **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/context-surfacing keep the 0.50/0.25/0.25 default.
144
+ - `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× … 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive on composite surfaces** (raw vector routes: pin = exact-tie precedence only).
145
+ - **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`, `memory_retrieve`'s composite modes, and context-surfacing keep the 0.50/0.25/0.25 default. (`vsearch` + `memory_retrieve` semantic/discovery use RAW cosine for non-recency queries as of v0.22.0 — no composite weights at all.)
146
146
  - **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
147
147
  - **Half-lives:** decision/deductive/preference/hub/antipattern = ∞ · project 120d · research 90d · problem/milestone/note 60d · conversation/progress 45d · handoff 30d (extend up to 3× for frequently-accessed).
148
148
 
@@ -162,7 +162,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
162
162
 
163
163
  ## Memory lifecycle (pin / snooze / forget — manual tools)
164
164
 
165
- - **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
165
+ - **`memory_pin`** (lifecycle retention + priority among relevance-equivalent results; +0.3 boost on composite surfaces, exact-tie precedence on raw vector routes) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
166
166
  - **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
167
167
  - **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
168
168
  - **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.
package/README.md CHANGED
@@ -542,8 +542,8 @@ Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
542
542
  | Tool | Description |
543
543
  |---|---|
544
544
  | `memory_retrieve` | **Preferred entry point.** Auto-classifies query and routes to optimal backend (query, intent_search, session_log, find_similar, or query_plan). Use instead of manually choosing a search tool. |
545
- | `search` | BM25 keyword search — for exact terms, config names, error codes, filenames. Composite scoring + co-activation boost + compact mode. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
546
- | `vsearch` | Vector semantic search — for conceptual/fuzzy matching when exact keywords are unknown. Composite scoring + co-activation boost + compact mode. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
545
+ | `search` | BM25 keyword search — for exact terms, config names, error codes, filenames. Composite scoring + co-activation boost + compact mode. **v0.23.0: the keyword relevance signal is real** — the exposed FTS score is a monotonic BM25 transform (a clamp bug flattened it to a constant 1.0 through v0.22.0, making ranking metadata-only). Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
546
+ | `vsearch` | Vector semantic search — for conceptual/fuzzy matching when exact keywords are unknown. **v0.22.0: non-recency queries rank by RAW cosine (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default)**; recency-intent queries keep composite. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
547
547
  | `query` | Full hybrid pipeline (BM25 + vector + rerank) — general-purpose when query type is unclear. WRONG for "why" questions (use `intent_search`) or cross-session queries (use `session_log`). Prefer `memory_retrieve` for auto-routing. Intent hint, strong-signal bypass, chunk dedup, candidateLimit, MMR diversity, compact mode. |
548
548
  | `get` | Retrieve single document by path or docid |
549
549
  | `multi_get` | Retrieve multiple docs by glob or comma-separated list |
@@ -594,8 +594,8 @@ Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
594
594
 
595
595
  | Tool | Description |
596
596
  |---|---|
597
- | `memory_forget` | Search → deactivate closest match (with audit trail) |
598
- | `memory_pin` | Pin a memory for +0.3 composite boost. USE PROACTIVELY when: user states a persistent constraint, makes an architecture decision, or corrects a misconception. Don't wait for curator — pin critical decisions immediately. |
597
+ | `memory_forget` | Search → deactivate closest match (with audit trail). Weak matches return a disambiguation list instead of acting (v0.23.0) |
598
+ | `memory_pin` | Pin a memory: lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on hook/`query`/`search` surfaces; exact-tie precedence on the raw vector routes). USE PROACTIVELY when: user states a persistent constraint, makes an architecture decision, or corrects a misconception. Don't wait for curator — pin critical decisions immediately. |
599
599
  | `memory_snooze` | Temporarily hide a memory from context surfacing until a date. USE PROACTIVELY when `<vault-context>` repeatedly surfaces irrelevant content — snooze for 30 days instead of ignoring it. |
600
600
  | `status` | Index health with content type distribution |
601
601
  | `reindex` | Trigger vault re-scan |
@@ -695,7 +695,7 @@ Content types are inferred from frontmatter or file path patterns. Half-lives ex
695
695
 
696
696
  **Frequency boost:** Documents with higher revision counts or duplicate counts get a durability signal: `freqSignal = (revisions - 1) × 2 + (duplicates - 1)`, `freqBoost = min(0.10, log1p(freqSignal) × 0.03)`. Revision count (content evolution) is weighted 2× vs duplicate count (ingest repetition). Capped at 10%.
697
697
 
698
- **Pin boost:** Pinned documents get +0.3 additive boost (capped at 1.0). Use `memory_pin` to pin critical memories.
698
+ **Pin boost:** Pin = lifecycle retention + priority among relevance-equivalent results. On composite surfaces pinned documents get a +0.3 additive boost (capped at 1.0); on the raw vector routes (v0.22.0) pin breaks exact raw-score ties only. Use `memory_pin` to pin critical memories.
699
699
 
700
700
  **Snooze:** Snoozed documents are filtered out of context surfacing until their snooze date. Use `memory_snooze` for temporary suppression.
701
701
 
@@ -791,7 +791,7 @@ Notes referenced by the agent during a session get boosted (`access_count++`). U
791
791
  | `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server URL. Without it, falls to `node-llama-cpp` (if allowed). |
792
792
  | `CLAWMEM_RERANK_API_KEY` | (none) | Bearer token for an authenticated remote reranker endpoint. Independent of the embed and LLM keys. |
793
793
  | `CLAWMEM_NO_LOCAL_MODELS` | `false` | Block `node-llama-cpp` from auto-downloading GGUF models. Set `true` for remote-only setups where you want fail-fast on unreachable endpoints. |
794
- | `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | `false` | **v0.21.0.** When `true`, the MCP direct tools (`search`, `vsearch`, `memory_retrieve`) score non-recency queries with the retrieval-tuned `query`-tool weights. Ships off the eval evidence covered only the hybrid `query` pipeline. YAML equivalent: `retrieval.mcp_direct_tuned_weights` (env wins). |
794
+ | `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | (superseded) | **No effect since v0.22.0** the direct-pipeline eval this knob was gated on measured tuned weights at 1/19 hit@1; the direct vector routes rank by raw cosine instead. Still parsed for backward compatibility; setting it logs a once-per-process warning. |
795
795
  | `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 consolidation merge-safety threshold when candidate and existing anchors align. Merges above this normalized 3-gram cosine score are allowed. |
796
796
  | `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety threshold — fallback when anchor sets are ambiguous. |
797
797
  | `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration before enabling the gate. |
package/SKILL.md CHANGED
@@ -92,9 +92,9 @@ All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculat
92
92
  | `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why" (use `intent_search`). |
93
93
  | `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
94
94
  | `profile` | User profile (static facts + dynamic context). |
95
- | `memory_pin` | +0.3 composite boost. Use PROACTIVELY for constraints, architecture decisions, corrections. |
95
+ | `memory_pin` | Lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on composite surfaces; exact-tie precedence on raw vector routes). Use PROACTIVELY for constraints, architecture decisions, corrections. |
96
96
  | `memory_snooze` | Use PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
97
- | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
97
+ | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. Weak matches return a disambiguation list instead of acting (v0.23.0). |
98
98
  | `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
99
99
  | `timeline` | Temporal neighborhood around a doc. Progressive disclosure: search → timeline → get. |
100
100
  | `memory_evolution_status` | How a doc's A-MEM metadata evolved over time. |
@@ -189,7 +189,7 @@ Query -> intent classification (WHY/WHEN/ENTITY/WHAT)
189
189
 
190
190
  ## Composite scoring (how ranking works)
191
191
 
192
- Applied automatically to all search results.
192
+ Applied on the composite surfaces: hooks, `query`, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. **v0.22.0: MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine instead** (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default); recency-intent queries keep composite everywhere. **v0.23.0:** `searchScore` on FTS surfaces is the monotonic `|bm25|/(1+|bm25|)` transform (it was a constant 1.0 through v0.22.0 due to a clamp bug — keyword relevance contributed zero ordering); FTS-transform scores and cosines are independent monotonic signals, not one calibrated scale.
193
193
 
194
194
  ```
195
195
  compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
@@ -198,8 +198,8 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
198
198
  - `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× penalty … 1.3× boost).
199
199
  - `coActivationBoost = 1 + min(coCount/10, 0.15)` (docs surfaced together get up to +15%).
200
200
  - Length normalization penalizes verbose entries (floor 30%); frequency boost capped at +10%.
201
- - **Pinned docs: +0.3 additive** (capped at 1.0).
202
- - **`query` tool (v0.13.0+):** non-recency queries use retrieval-tuned **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/`context-surfacing` keep the 0.50/0.25/0.25 default.
201
+ - **Pinned docs: +0.3 additive on composite surfaces** (capped at 1.0); on the raw vector routes pin = exact-tie precedence only.
202
+ - **`query` tool (v0.13.0+):** non-recency queries use retrieval-tuned **0.70·search + 0.15·recency + 0.15·confidence**. `search`, `memory_retrieve`'s composite modes, and `context-surfacing` keep the 0.50/0.25/0.25 default. (`vsearch` + `memory_retrieve` semantic/discovery use RAW cosine for non-recency queries as of v0.22.0 — no composite weights at all.)
203
203
  - **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
204
204
 
205
205
  **Content-type half-lives:** decision / deductive / preference / hub / antipattern = ∞ (never decay) · project 120d · research 90d · problem / milestone / note 60d · conversation / progress 45d · handoff 30d. Half-lives extend up to 3× for frequently-accessed memories. Attention decay: non-durable types (handoff, progress, conversation, note, project) lose 5% confidence/week without access; decision / deductive / preference / hub / research / antipattern are exempt.
@@ -210,7 +210,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
210
210
 
211
211
  ## Memory lifecycle (pin / snooze / forget — manual tools)
212
212
 
213
- - **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist across sessions. Do NOT pin routine/session-specific items.
213
+ - **`memory_pin`** (lifecycle retention + priority among relevance-equivalent results; +0.3 boost on composite surfaces, exact-tie precedence on raw vector routes) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist across sessions. Do NOT pin routine/session-specific items.
214
214
  - **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
215
215
  - **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
216
216
  - **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.
@@ -56,7 +56,7 @@ Each indexed file becomes a document with:
56
56
  | `content_type` | Auto-detected: decision, deductive, preference, note, handoff, conversation, progress, research, hub, antipattern, project, milestone, problem |
57
57
  | `quality_score` | 0.0-1.0 based on length, structure, headings, lists, decision keywords, frontmatter |
58
58
  | `confidence` | Starts at 0.5, adjusted by contradiction detection and feedback |
59
- | `pinned` | Manual boost flag (+0.3 composite score) |
59
+ | `pinned` | Lifecycle retention + priority among relevance-equivalent results (+0.3 on composite surfaces; exact-tie precedence on the raw vector routes) |
60
60
  | `snoozed_until` | Temporarily hidden from context surfacing |
61
61
 
62
62
  ## Fragments
@@ -1,6 +1,6 @@
1
1
  # Composite Scoring
2
2
 
3
- Every search result in ClawMem is scored using a composite formula that blends multiple signals beyond raw search relevance.
3
+ Composite scoring blends multiple signals beyond raw search relevance. **It applies to the hook pipeline (context surfacing), the `query` hybrid pipeline, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes.** Since v0.22.0 the direct vector routes — MCP `vsearch` and `memory_retrieve` semantic/discovery — do NOT use it for non-recency queries: they rank by raw vector cosine, with metadata (including pin) breaking exact score ties only, and report `scoreBasis: "vector-cosine"`. That split is measured, not aesthetic: on a judged set against the live vault, raw cosine ranked 16/19 targets #1 (MRR 0.912) while this composite ranked 1/19 (MRR 0.307) and pushed 14/19 correct answers below the old minScore floor — in the compressed-high similarity band of modern embedding models, every multiplier below is larger than the raw margins separating right answers from wrong ones. Recency-intent queries keep composite (the Recency column) on every route that had it.
4
4
 
5
5
  ## Formula
6
6
 
@@ -19,10 +19,12 @@ compositeScore = (w_search * searchScore + w_recency * recencyScore + w_confiden
19
19
 
20
20
  Recency intent is detected automatically when queries contain "latest", "recent", "last session", etc. — and **takes precedence over the `query`-tool weights**: a recency-phrased `query` call still uses the Recency-intent column.
21
21
 
22
- The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. `search`, `vsearch`, `memory_retrieve`, and the context-surfacing hook **keep the Normal column by default**. Since v0.21.0 a config knob `retrieval.mcp_direct_tuned_weights: true` (or env `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS=true`)switches the three MCP direct tools' non-recency scoring to the `query`-tool column; it ships **off** because the eval evidence covered only the hybrid `query` pipeline, and the flip is gated on a direct-pipeline eval (see [configuration](../reference/configuration.md)).
22
+ The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. `search`, the composite `memory_retrieve` modes, and the context-surfacing hook **keep the Normal column**. The v0.21.0 `retrieval.mcp_direct_tuned_weights` knob is **superseded as of v0.22.0 and has no effect** — the direct-pipeline eval it was gated on measured tuned weights at 1/19 hit@1, and the direct vector routes moved to raw ordering instead. The key is still parsed (a once-per-process warning is logged if set) so existing configs don't break.
23
23
 
24
24
  Threshold note: EOS-anchored last-token embedding models (e.g. zembed-1 served correctly) produce a **compressed-high similarity band** — unrelated pairs sit near ~0.4 rather than ~0.2. Relative ordering is what matters; treat absolute `minScore` cutoffs as model-dependent rather than universal.
25
25
 
26
+ `searchScore` provenance (v0.23.0): on FTS/BM25 surfaces the input is the monotonic transform `|bm25|/(1+|bm25|)` of FTS5's negative-is-better `bm25()` (through v0.22.0 a clamp bug flattened it to a constant 1.0, which made composite ranking on those surfaces metadata-only); on vector surfaces it is the raw cosine. The two are independent monotonic signals on an uncalibrated common range — composite blends them where pools mix (hooks, REST hybrid), which removes the old unconditional FTS-over-vector dominance but does not make the channels numerically comparable.
27
+
26
28
  ## Signal breakdown
27
29
 
28
30
  ### Search score (0.0 - 1.0)
@@ -92,13 +94,13 @@ coActivationBoost = 1 + min(coCount / 10, 0.15)
92
94
 
93
95
  Documents frequently surfaced together in the same session get up to 15% boost.
94
96
 
95
- **Where co-activation is applied depends on the caller.** MCP tools (`query`, `search`, `vsearch`) pass a co-activation function into `applyCompositeScoring()`, so the boost is part of the composite score used for ranking and `minScore` filtering. The context-surfacing hook does NOT pass co-activation into composite scoring — instead it applies a separate spreading-activation step *after* adaptive threshold filtering. This means the hook's threshold decisions are based on scores without co-activation, and co-activation only boosts results that already passed the threshold. This is intentional: it prevents relationship boosts from rescuing otherwise-weak results into the surfaced set.
97
+ **Where co-activation is applied depends on the caller.** The composite MCP surfaces (`query`, `search`, and `vsearch` on recency-intent queries only) pass a co-activation function into `applyCompositeScoring()`, so the boost is part of the composite score used for ranking and — on `vsearch` and `search`, the tools that expose `minScore` — for `minScore` filtering. On `vsearch`'s non-recency (raw) regime, co-activation contributes only to the exact-tie key, never to ranking or filtering. The context-surfacing hook does NOT pass co-activation into composite scoring — instead it applies a separate spreading-activation step *after* adaptive threshold filtering. This means the hook's threshold decisions are based on scores without co-activation, and co-activation only boosts results that already passed the threshold. This is intentional: it prevents relationship boosts from rescuing otherwise-weak results into the surfaced set.
96
98
 
97
99
  ## Additional modifiers
98
100
 
99
101
  ### Pin boost
100
102
 
101
- Pinned documents receive +0.3 additive boost (capped at 1.0 total).
103
+ On composite surfaces, pinned documents receive a +0.3 additive boost (capped at 1.0 total). Pin's contract is **lifecycle retention + prioritization among relevance-equivalent results** — on the raw vector routes (v0.22.0) it therefore breaks exact raw-score ties only and never lifts a document over a more relevant one.
102
104
 
103
105
  ### Length normalization
104
106
 
@@ -1,6 +1,6 @@
1
1
  # Upgrading ClawMem
2
2
 
3
- Guide for upgrading between released versions. Current: **v0.21.0**.
3
+ Guide for upgrading between released versions. Current: **v0.23.0**.
4
4
 
5
5
  ClawMem upgrades are designed to be drop-in: pull the new version, restart any long-lived processes, and the SQLite schema auto-migrates on first open. This guide documents per-version specifics for upgrades that have additional considerations beyond the quick path below.
6
6
 
@@ -59,6 +59,31 @@ docker compose up -d reranker # /v1/rerank on :8090
59
59
 
60
60
  ---
61
61
 
62
+ ## v0.23.0: monotonic BM25 exposed score
63
+
64
+ No migration command, no schema change, no re-embed. Restart long-lived processes per the quick path.
65
+
66
+ **Behavior change — the exposed FTS score is a real relevance signal.** Through v0.22.0 a clamp bug flattened every FTS result's `score` to the constant 1.0, so ranking on the BM25 surfaces (`search`, REST keyword mode, CLI search, `memory_retrieve` keyword and its semantic-mode FTS fallback, hook FTS lanes) was effectively metadata-only. The score is now `|bm25|/(1+|bm25|)` — bounded [0,1), higher is better. What to expect after upgrading:
67
+
68
+ - `search` results reorder toward keyword relevance; reported scores drop from the old flat values and vary per hit. If a workflow compared `search` scores against a hardcoded cutoff tuned to the constant-1.0 era, re-tune it (the composite floor semantics of `minScore` are unchanged; the observed values shifted).
69
+ - The `query` pipeline's strong-signal bypass actually fires now on unambiguous keyword queries (skipping LLM expansion, which makes those calls faster) — and no longer fires on a lone weak match.
70
+ - `memory_forget` targeting is stricter: weak keyword matches return a disambiguation list instead of auto-selecting. If a script relied on forget acting on any single match, it must now pass a more specific query or a path.
71
+ - `clawmem doctor`/curator's BM25 probe reports honestly — a near-empty vault may now show a degraded BM25 probe where it previously passed vacuously.
72
+
73
+ ---
74
+
75
+ ## v0.22.0: raw-cosine ranking on the direct vector routes
76
+
77
+ No migration command, no schema change, no re-embed. Restart long-lived processes per the quick path.
78
+
79
+ **Behavior change — MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine.** Reported scores on those routes are raw cosine (`scoreBasis: "vector-cosine"`), not composite values — expect a different numeric scale (unrelated results sit near ~0.4–0.55 on compressed-band embedding models, not near 0). `vsearch`'s `minScore` filters that raw scale and no longer defaults to 0.3 — omitted means no filter. If a workflow passed `minScore` tuned to composite values, re-tune it to your embedding model's band or omit it. Recency-phrased queries ("latest…", "recently…") behave exactly as before on every route.
80
+
81
+ **Pinned documents** no longer float above more relevant results on those two routes — pin now means lifecycle retention plus winning exact-score ties. Hooks, `query`, and `search` keep the +0.3 composite pin boost.
82
+
83
+ **`retrieval.mcp_direct_tuned_weights` / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` no longer has any effect** (superseded by its own gating eval). Configs that set it keep working; a once-per-process warning is logged — remove the key at your convenience.
84
+
85
+ ---
86
+
62
87
  ## v0.21.0: MCP internal-collection exclusion + embedding-geometry canary
63
88
 
64
89
  No migration command required. Two things to know:
@@ -71,7 +96,7 @@ No migration command required. Two things to know:
71
96
 
72
97
  **The canary baseline seeds itself.** The first `clawmem embed` run after upgrade (including a timer-fired one) probes the embedding server with a pair-separation battery and persists a first-healthy baseline; subsequent runs alert relative to it, and a broken-geometry server now aborts `embed --force` BEFORE anything is cleared. `--force-geometry` (proceed despite a failed probe; the vault is flagged tainted until a verified rebuild) and `--force --recalibrate-canary` (replace the baseline after a deliberate model/server change) are operator overrides, not upgrade steps.
73
98
 
74
- **Opt-in knob:** `retrieval.mcp_direct_tuned_weights` (config) / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` (env), default `false` — scores the MCP direct tools' non-recency queries with the retrieval-tuned `query`-tool weights. Leave it off unless you have direct-pipeline eval evidence for your vault.
99
+ **Opt-in knob:** `retrieval.mcp_direct_tuned_weights` (config) / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` (env), default `false` — scored the MCP direct tools' non-recency queries with the retrieval-tuned `query`-tool weights. *(Superseded in v0.22.0 the knob no longer has any effect; see the v0.22.0 section above.)*
75
100
 
76
101
  Upgrades from v0.13 → v0.20 shipped no steps beyond the [quick path](#quick-path) — schema changes auto-apply; see [RELEASE_NOTES.md](../../RELEASE_NOTES.md) for what each version changed.
77
102
 
@@ -79,6 +79,8 @@ When the top BM25 hit scores >= 0.85 and the gap to the second hit is >= 0.15, t
79
79
 
80
80
  The bypass is disabled when `intent` is provided — intent implies the query is ambiguous, so keyword confidence alone is insufficient.
81
81
 
82
+ Functional as of v0.23.0: through v0.22.0 a clamp bug flattened every FTS score to a constant 1.0, so with two or more hits the gap was always 0 (the bypass never fired) and with exactly one hit the gap was always 1.0 (it fired even on a garbage match). The scores are now the monotonic `|bm25|/(1+|bm25|)` transform — 0.85 corresponds to |bm25| ≥ 5.67 — and the check itself lives in one shared helper (`hasStrongFtsSignal`) used by both this pipeline and the CLI `query` command. Threshold tuning is deliberately deferred to a judged query-pipeline A/B (BACKLOG 49.3).
83
+
82
84
  ## Query expansion
83
85
 
84
86
  The LLM generates lex (keyword), vec (semantic), and hyde (hypothetical answer) variants of the query, each carried as a typed `ExpandedQuery`. Variants are **routed by type**: `lex` expansions are searched on BM25 only, `vec` and `hyde` on vector only. The original query is the only leg that fans out to *both* backends. All legs are searched in parallel and fused; the original query's two lists receive 2x weight in RRF, ensuring it anchors the ranking.
@@ -6,7 +6,7 @@ ClawMem is an open-source memory engine for Claude Code and AI agents. It runs o
6
6
 
7
7
  - **Indexes** markdown documents into a local SQLite vault with full-text search (BM25) and vector embeddings
8
8
  - **Retrieves** relevant context automatically via hooks or on-demand via MCP tools / REST API
9
- - **Scores** results using composite scoring (relevance, recency, confidence, quality, co-activation)
9
+ - **Scores** results per route: raw vector cosine on the direct vector tools (v0.22.0), composite scoring (relevance, recency, confidence, quality, co-activation) on the hook and hybrid pipelines
10
10
  - **Tracks** decisions, handoffs, and session history across conversations
11
11
  - **Traverses** causal and semantic graphs to answer "why" and "what led to" questions
12
12
 
@@ -38,7 +38,7 @@ Full provider matrix and behavior: [../guides/cloud-embedding.md](../guides/clou
38
38
  |---|---|---|
39
39
  | `CLAWMEM_PROFILE` | `balanced` | `speed` / `balanced` / `deep`. Sets the kept-score ratio (65% / 55% / 45%), vector timeout, max results, and the `factsTokens` sub-budget. Only `deep` adds query expansion + reranking to the hook path. `speed` makes hooks BM25-only (sub-500ms). |
40
40
  | `CLAWMEM_NUDGE_INTERVAL` | `15` | Prompts between lifecycle tool use before a `<vault-nudge>` is injected. `0` to disable. |
41
- | `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | `false` | When `true`, the MCP direct tools (`search`, `vsearch`, `memory_retrieve`) score non-recency queries with the retrieval-tuned `query`-tool weights (search 0.70) instead of the Normal weights. Ships **off**: the eval evidence covered only the hybrid `query` pipeline; flipping the default is gated on a direct-pipeline eval. YAML equivalent in `config.yaml`: `retrieval.mcp_direct_tuned_weights: true` (env wins). |
41
+ | `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | (superseded) | **No effect since v0.22.0.** The direct-pipeline eval this knob was gated on measured tuned weights at 1/19 hit@1; the direct vector routes now rank by raw cosine instead (see [mcp-tools](mcp-tools.md) Scoring regimes). Still parsed for backward compatibility setting it (env or `retrieval.mcp_direct_tuned_weights` in `config.yaml`) logs a once-per-process warning. |
42
42
 
43
43
  The context-surfacing hook `timeout` is **not** an env var — it lives in `~/.claude/settings.json` (8s default). See [../troubleshooting.md](../troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
44
44
 
@@ -6,6 +6,10 @@ Complete reference for ClawMem's MCP server tools. These let AI agents search, r
6
6
 
7
7
  **Internal-collection visibility (v0.21.0):** the retrieval tools `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` exclude the system-internal `_clawmem` collection (observations/deductions/handoffs) by default. Opt-ins: pass `includeInternal: true` (all six tools), or — on the tools that expose a `collection` parameter (`search`, `vsearch`, `query`) — name `_clawmem` explicitly in the filter. `find_similar` auto-includes internal results when the REFERENCE document is itself internal. `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are NOT filtered — system memory is their substrate by design.
8
8
 
9
+ **Scoring regimes (v0.22.0):** the direct vector routes — `vsearch` and `memory_retrieve`'s semantic/discovery modes — rank non-recency queries by RAW vector cosine. Their `structuredContent` carries `scoreBasis: "vector-cosine"`; raw cosine values are specific to the embedding model that produced them and are NOT comparable across models or to composite scores. Document metadata — including pin — participates only inside groups of exactly-equal raw scores. On these routes `minScore` filters the raw score and has NO default (omitted = no filter; an explicit `0` is honored). Recency-intent queries ("latest…", "recently…", "yesterday…") keep the composite regime with its 0.3 default floor and report `scoreBasis: "composite"`, as do `search`, `query`, `query_plan`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. `find_similar` has always ranked by raw cosine. Rationale: on the measured vault, raw cosine ranked 16/19 judged targets #1 (MRR 0.912) while the composite stack ranked 1/19 and filtered 14/19 out entirely.
10
+
11
+ **FTS score provenance (v0.23.0):** the raw `score` on BM25/FTS results is the monotonic transform `|bm25|/(1+|bm25|)` of FTS5's negative-is-better `bm25()` value — bounded [0,1), higher is better, stable across queries. (Through v0.22.0 a clamp bug flattened it to a constant 1.0, so composite ranking on FTS surfaces was effectively metadata-only and score-threshold gates never filtered.) FTS-transform scores and vector cosines are **independent monotonic signals, not a calibrated common scale** — compare within a channel, not across channels.
12
+
9
13
  **Degraded vector results (v0.21.0):** under default exclusion the vector scan escalates its depth to fill `limit` with allowed documents, up to a hard cap. When the cap prevents an exhaustive scan and the result is under-filled, `structuredContent` carries `degraded: true` with `degradedReason`: `"excluded-dominant"` (distinct excluded docs account for the shortfall — the guidance line suggests `includeInternal: true` or a refined query) or `"cap-truncation"` (shortfall driven by fragment dedup, neutral guidance). Multi-leg routes (`query`, `query_plan`, `memory_retrieve` complex mode) aggregate `degraded = any(leg)` and list per-leg reasons in `structuredContent.degradedLegs`; single-vector routes (`vsearch`, `find_similar`, `memory_retrieve`'s other modes) report the flat `degraded` + `degradedReason` pair. A small vault whose whole index is scanned without hitting the cap returns a plain short list with NO marker.
10
14
 
11
15
  ### memory_retrieve
@@ -47,25 +51,29 @@ BM25 strong-signal bypass: skips expansion when top BM25 hit >= 0.85 with gap >=
47
51
 
48
52
  ### search
49
53
 
50
- BM25 only. Zero GPU cost.
54
+ BM25 only. Zero GPU cost. Ranking is composite (see the score-provenance note above); the underlying keyword relevance signal is the monotonic BM25 transform introduced in v0.23.0.
51
55
 
52
56
  | Param | Type | Default | Description |
53
57
  |-------|------|---------|-------------|
54
58
  | `query` | string | required | Search query |
55
59
  | `limit` | number | 10 | Max results |
60
+ | `minScore` | number | 0 | Composite-score floor (values shifted in v0.23.0 — the searchScore input is now a real relevance signal, not a constant) |
56
61
  | `compact` | boolean | true | Compact output |
57
62
  | `collection` | string | — | Filter by collection |
58
63
  | `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
59
64
  | `vault` | string | — | Named vault |
60
65
 
66
+ Score fields: compact results report the rounded **composite** score; non-compact results carry both `score` (raw BM25 transform) and `compositeScore`.
67
+
61
68
  ### vsearch
62
69
 
63
- Vector only. Semantic similarity.
70
+ Vector only. Semantic similarity. Non-recency queries rank by RAW cosine (`scoreBasis: "vector-cosine"`, v0.22.0); recency-intent queries use the composite regime.
64
71
 
65
72
  | Param | Type | Default | Description |
66
73
  |-------|------|---------|-------------|
67
74
  | `query` | string | required | Search query |
68
75
  | `limit` | number | 10 | Max results |
76
+ | `minScore` | number | — | Raw-cosine floor on non-recency queries (no default — omitted means no filter; explicit `0` honored). Recency-intent queries keep the composite-scale 0.3 default. |
69
77
  | `compact` | boolean | true | Compact output |
70
78
  | `collection` | string | — | Filter by collection |
71
79
  | `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
@@ -178,7 +186,7 @@ Recent session history with handoffs and file changes.
178
186
 
179
187
  ### memory_pin
180
188
 
181
- Pin a memory for permanent prioritization (+0.3 composite boost).
189
+ Pin a memory: lifecycle retention plus prioritization among relevance-equivalent results. On composite surfaces (hooks, `query`, `search`) pinned docs get the +0.3 composite boost; on the raw vector routes (v0.22.0) pin wins exact raw-score ties but never overrides a relevance difference.
182
190
 
183
191
  | Param | Type | Default | Description |
184
192
  |-------|------|---------|-------------|
@@ -208,6 +216,8 @@ Permanently deactivate a memory.
208
216
 
209
217
  **Search behavior (v0.2.6+, all three tools):** Query matching cascades through four strategies: exact path match → BM25 full-text → title-token overlap → vector similarity. This prevents "No matching memory found" errors when the document exists but BM25 fails to match (e.g., too many AND'd terms). Path-like queries (containing `/` or ending in `.md`) try direct path matching first. `memory_forget` requires higher confidence to act — ambiguous matches return candidates instead of mutating.
210
218
 
219
+ **Targeting confidence (v0.23.0):** the confidence gate (`score ≥ 0.7`, or a ≥ 0.2 gap to the runner-up when more than one candidate exists) is now live for BM25 candidates — through v0.22.0 every FTS candidate carried a constant score of 1.0, so `memory_forget` treated ANY keyword match as high-confidence and auto-selected it. Weak matches — including a lone weak match — now return the candidate list for disambiguation instead of acting. Stricter, safer targeting for a destructive operation.
220
+
211
221
  ## Lifecycle
212
222
 
213
223
  ### lifecycle_status
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "clawmem",
3
- "version": "0.21.0",
3
+ "version": "0.23.0",
4
4
  "description": "On-device memory layer for AI agents. Claude Code, OpenClaw, and Hermes. Hooks + MCP server + hybrid RAG search.",
5
5
  "type": "module",
6
6
  "bin": {
package/src/clawmem.ts CHANGED
@@ -58,7 +58,7 @@ import { formatSearchResults, type OutputFormat } from "./formatter.ts";
58
58
  import { indexCollection, parseDocument } from "./indexer.ts";
59
59
  import { detectBeadsProject } from "./beads.ts";
60
60
  import { applyCompositeScoring, hasRecencyIntent, type EnrichedResult } from "./memory.ts";
61
- import { enrichResults, reciprocalRankFusion, toRanked, type RankedResult } from "./search-utils.ts";
61
+ import { enrichResults, reciprocalRankFusion, toRanked, hasStrongFtsSignal, type RankedResult } from "./search-utils.ts";
62
62
  import { splitDocument } from "./splitter.ts";
63
63
  import { getProfile, updateProfile, isProfileStale } from "./profile.ts";
64
64
  import { regenerateAllDirectoryContexts } from "./directory-context.ts";
@@ -1095,9 +1095,7 @@ async function cmdQuery(args: string[]) {
1095
1095
 
1096
1096
  // Step 1: BM25 for strong signal check
1097
1097
  const ftsResults = s.searchFTS(query, 20);
1098
- const topScore = ftsResults[0]?.score ?? 0;
1099
- const secondScore = ftsResults[1]?.score ?? 0;
1100
- const strongSignal = topScore >= 0.85 && (topScore - secondScore) >= 0.15;
1098
+ const strongSignal = hasStrongFtsSignal(ftsResults);
1101
1099
 
1102
1100
  // Step 2: Query expansion (skip if strong BM25 signal). expandQuery now returns
1103
1101
  // typed ExpandedQuery[] (lex/vec/hyde) — no more brittle string re-parsing, and
package/src/config.ts CHANGED
@@ -54,11 +54,11 @@ export interface LifecyclePolicy {
54
54
 
55
55
  export interface RetrievalConfig {
56
56
  /**
57
- * Option-B knob (VSEARCH-TRUST-HARDENING (a)): when true, the MCP direct tools
58
- * (search / vsearch / memory_retrieve) score non-recency queries with the
59
- * retrieval-tuned QUERY_WEIGHTS instead of DEFAULT_WEIGHTS. Default FALSE
60
- * the flip is gated on a direct-pipeline eval (the n=199 evidence covered only
61
- * the hybrid `query` pipeline).
57
+ * SUPERSEDED as of v0.22.0 and no longer consumed anywhere: the direct-pipeline
58
+ * eval this knob was gated on measured tuned weights at 1/19 hit@1 (BACKLOG
59
+ * Source 48), and the direct vector routes now rank by raw similarity instead
60
+ * (VSEARCH-RAW-PRIMARY-DESIGN.md). Still parsed so existing configs don't break;
61
+ * setting it warns once per process.
62
62
  */
63
63
  mcp_direct_tuned_weights: boolean;
64
64
  }
@@ -126,6 +126,8 @@ export function getActiveProfile(): ProfileConfig {
126
126
  // ---------------------------------------------------------------------------
127
127
 
128
128
  let _cachedConfig: ClawMemConfig | null = null;
129
+ // Once-per-process compatibility warning for the superseded tuned-weights knob (v0.22.0).
130
+ let _warnedTunedWeightsKnob = false;
129
131
 
130
132
  /**
131
133
  * Load vault configuration from env vars and config file.
@@ -186,13 +188,18 @@ export function loadVaultConfig(): ClawMemConfig {
186
188
  };
187
189
  }
188
190
 
189
- // 4. Retrieval knobs (optional). Env override for tests / staged rollout.
191
+ // 4. Retrieval knobs (optional). Parsed for backward compatibility only
192
+ // mcp_direct_tuned_weights is superseded as of v0.22.0 and has no effect.
190
193
  let retrieval: RetrievalConfig | undefined;
191
194
  const envTuned = process.env.CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS;
192
- const yamlTuned = parsedYaml?.retrieval && typeof parsedYaml.retrieval === "object"
193
- ? parsedYaml.retrieval.mcp_direct_tuned_weights === true
194
- : false;
195
+ const yamlHasTuned = !!(parsedYaml?.retrieval && typeof parsedYaml.retrieval === "object"
196
+ && parsedYaml.retrieval.mcp_direct_tuned_weights !== undefined);
197
+ const yamlTuned = yamlHasTuned ? parsedYaml.retrieval.mcp_direct_tuned_weights === true : false;
195
198
  retrieval = { mcp_direct_tuned_weights: envTuned !== undefined ? envTuned === "true" : yamlTuned };
199
+ if ((envTuned !== undefined || yamlHasTuned) && !_warnedTunedWeightsKnob) {
200
+ _warnedTunedWeightsKnob = true;
201
+ console.warn("[clawmem] retrieval.mcp_direct_tuned_weights / CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS is superseded as of v0.22.0 and has no effect — the direct vector routes rank by raw similarity.");
202
+ }
196
203
 
197
204
  _cachedConfig = { vaults, lifecycle, retrieval };
198
205
  return _cachedConfig;
package/src/mcp.ts CHANGED
@@ -33,7 +33,8 @@ import {
33
33
  type EnrichedResult,
34
34
  type CoActivationFn,
35
35
  } from "./memory.ts";
36
- import { enrichResults, reciprocalRankFusion, toRanked, blendRerank, type RankedResult } from "./search-utils.ts";
36
+ import { enrichResults, reciprocalRankFusion, toRanked, blendRerank, hasStrongFtsSignal, attachRrfScores, type RankedResult } from "./search-utils.ts";
37
+ import { selectScoringRegime, rankRawPrimary, VECTOR_SCORE_BASIS, COMPOSITE_SCORE_BASIS } from "./scoring-regime.ts";
37
38
  import { applyMMRDiversity } from "./mmr.ts";
38
39
  import { indexCollection, type IndexStats } from "./indexer.ts";
39
40
  import { listCollections } from "./collections.ts";
@@ -140,12 +141,6 @@ function degradedGuidanceText(legs: DegradedLeg[]): string {
140
141
  : "Note: vector results truncated at the scan cap.";
141
142
  }
142
143
 
143
- // Option-B weights knob (design (a)): default-off; flips search/vsearch/memory_retrieve
144
- // non-recency scoring to the retrieval-tuned weights only when explicitly enabled.
145
- function mcpDirectWeightsOption(): { weights: typeof QUERY_WEIGHTS } | undefined {
146
- return loadVaultConfig().retrieval?.mcp_direct_tuned_weights ? { weights: QUERY_WEIGHTS } : undefined;
147
- }
148
-
149
144
  function classifyRetrievalMode(query: string): "keyword" | "semantic" | "causal" | "timeline" | "discovery" | "complex" | "hybrid" {
150
145
  const q = query.toLowerCase();
151
146
 
@@ -335,11 +330,7 @@ This is the recommended entry point for ALL memory queries.`,
335
330
  } catch (e) { rethrowIfFatalVectorError(e); /* else: no vectors */ }
336
331
  const rrfWeights = intent.intent === 'WHY' ? [1.0, 1.5] : intent.intent === 'WHEN' ? [1.5, 1.0] : [1.0, 1.0];
337
332
  const fusedRanked = reciprocalRankFusion([bm25Results.map(toRanked), vecResults.map(toRanked)], rrfWeights);
338
- const allSearch = [...bm25Results, ...vecResults];
339
- let fused: SearchResult[] = fusedRanked.map(fr => {
340
- const orig = allSearch.find(r => r.filepath === fr.file);
341
- return orig ? { ...orig, score: fr.score } : null;
342
- }).filter((r): r is SearchResult => r !== null);
333
+ let fused: SearchResult[] = attachRrfScores(fusedRanked, [...bm25Results, ...vecResults]);
343
334
 
344
335
  if (intent.intent === 'WHY' || intent.intent === 'ENTITY') {
345
336
  try {
@@ -388,7 +379,7 @@ This is the recommended entry point for ALL memory queries.`,
388
379
  }
389
380
 
390
381
  const enriched = enrichResults(store, fused, query);
391
- const scored = applyCompositeScoring(enriched, query, undefined, mcpDirectWeightsOption()).slice(0, lim);
382
+ const scored = applyCompositeScoring(enriched, query).slice(0, lim);
392
383
  const items = scored.map(r => ({
393
384
  docid: `#${r.docid}`, path: r.displayPath, title: r.title,
394
385
  score: Math.round(r.compositeScore * 100) / 100,
@@ -426,7 +417,7 @@ This is the recommended entry point for ALL memory queries.`,
426
417
  const seen = new Set<string>();
427
418
  const deduped = allResults.filter(r => { if (seen.has(r.filepath)) return false; seen.add(r.filepath); return true; });
428
419
  const enriched = enrichResults(store, deduped, query);
429
- const scored = applyCompositeScoring(enriched, query, undefined, mcpDirectWeightsOption()).slice(0, lim);
420
+ const scored = applyCompositeScoring(enriched, query).slice(0, lim);
430
421
  const items = scored.map(r => ({
431
422
  docid: `#${r.docid}`, path: r.displayPath, title: r.title,
432
423
  score: Math.round(r.compositeScore * 100) / 100,
@@ -442,12 +433,16 @@ This is the recommended entry point for ALL memory queries.`,
442
433
  // --- Keyword / Semantic / Discovery / Hybrid modes ---
443
434
  let results: SearchResult[] = [];
444
435
  let singleLegDegraded: DegradedLeg | undefined;
436
+ // semantic/discovery: the raw regime applies ONLY to results the vector leg actually
437
+ // served — the FTS fallback's scores are not cosine, so it keeps composite scoring.
438
+ let vectorLegServed = false;
445
439
  if (effectiveMode === "keyword") {
446
440
  results = store.searchFTS(query, lim, undefined, undefined, undefined, excl);
447
441
  } else if (effectiveMode === "semantic" || effectiveMode === "discovery") {
448
442
  try {
449
443
  const det = await store.searchVecDetailed(query, DEFAULT_EMBED_MODEL, lim, { excludeCollections: excl });
450
444
  results = det.results;
445
+ vectorLegServed = true;
451
446
  if (det.degraded && det.degradedReason) singleLegDegraded = { leg: `${effectiveMode}:vector`, reason: det.degradedReason };
452
447
  } catch (e) { rethrowIfFatalVectorError(e); results = store.searchFTS(query, lim, undefined, undefined, undefined, excl); }
453
448
  } else {
@@ -461,18 +456,24 @@ This is the recommended entry point for ALL memory queries.`,
461
456
  } catch (e) { rethrowIfFatalVectorError(e); /* */ }
462
457
  if (vec.length > 0) {
463
458
  const fusedRanked = reciprocalRankFusion([bm25.map(toRanked), vec.map(toRanked)], [1.0, 1.0]);
464
- const allSearch = [...bm25, ...vec];
465
- results = fusedRanked.map(fr => {
466
- const orig = allSearch.find(r => r.filepath === fr.file);
467
- return orig ? { ...orig, score: fr.score } : null;
468
- }).filter((r): r is SearchResult => r !== null);
459
+ results = attachRrfScores(fusedRanked, [...bm25, ...vec]);
469
460
  } else {
470
461
  results = bm25;
471
462
  }
472
463
  }
473
464
 
474
465
  const enriched = enrichResults(store, results, query);
475
- const scored = applyCompositeScoring(enriched, query, undefined, mcpDirectWeightsOption()).slice(0, lim);
466
+ // v0.22.0 two-regime scoring (VSEARCH-RAW-PRIMARY-DESIGN.md R1/R4): vector-served
467
+ // semantic/discovery results on non-recency queries rank by raw cosine (metadata
468
+ // breaks exact ties only). Keyword/hybrid modes and the FTS fallback keep composite —
469
+ // their scores are not cosine, so the raw contract cannot hold there.
470
+ const rawRegime = vectorLegServed
471
+ && (effectiveMode === "semantic" || effectiveMode === "discovery")
472
+ && selectScoringRegime(query) === "raw";
473
+ const retrieveScoreBasis = rawRegime ? VECTOR_SCORE_BASIS : COMPOSITE_SCORE_BASIS;
474
+ const scored = rawRegime
475
+ ? rankRawPrimary(enriched, query).slice(0, lim)
476
+ : applyCompositeScoring(enriched, query).slice(0, lim);
476
477
  const modeDegraded = singleLegDegraded ? { degraded: true as const, degradedReason: singleLegDegraded.reason } : {};
477
478
  const modeDegradedNote = singleLegDegraded ? `\n${degradedGuidanceText([singleLegDegraded])}` : "";
478
479
  if (compact) {
@@ -483,7 +484,7 @@ This is the recommended entry point for ALL memory queries.`,
483
484
  }));
484
485
  return {
485
486
  content: [{ type: "text", text: `[routed: ${effectiveMode}] ${formatSearchSummary(items.map(i => ({ ...i, file: i.path, compositeScore: i.score, context: null })), query)}${modeDegradedNote}` }],
486
- structuredContent: { mode: effectiveMode, results: items, ...modeDegraded },
487
+ structuredContent: { mode: effectiveMode, results: items, scoreBasis: retrieveScoreBasis, ...modeDegraded },
487
488
  };
488
489
  }
489
490
  const items: SearchResultItem[] = scored.map(r => {
@@ -497,7 +498,7 @@ This is the recommended entry point for ALL memory queries.`,
497
498
  });
498
499
  return {
499
500
  content: [{ type: "text", text: `[routed: ${effectiveMode}] ${formatSearchSummary(items, query)}${modeDegradedNote}` }],
500
- structuredContent: { mode: effectiveMode, results: items, ...modeDegraded },
501
+ structuredContent: { mode: effectiveMode, results: items, scoreBasis: retrieveScoreBasis, ...modeDegraded },
501
502
  };
502
503
  }
503
504
  );
@@ -585,7 +586,7 @@ This is the recommended entry point for ALL memory queries.`,
585
586
 
586
587
  const coFn = (path: string) => store.getCoActivated(path);
587
588
  const enriched = enrichResults(store, results, query);
588
- const scored = applyCompositeScoring(enriched, query, coFn, mcpDirectWeightsOption())
589
+ const scored = applyCompositeScoring(enriched, query, coFn)
589
590
  .filter(r => r.compositeScore >= (minScore || 0));
590
591
 
591
592
  if (compact) {
@@ -631,7 +632,7 @@ This is the recommended entry point for ALL memory queries.`,
631
632
  inputSchema: {
632
633
  query: z.string().describe("Natural language query"),
633
634
  limit: z.number().optional().default(10),
634
- minScore: z.number().optional().default(0.3),
635
+ minScore: z.number().optional().describe("Score floor. Non-recency queries filter the RAW cosine score (default: no filter; explicit 0 honored; cosine values are embedding-model-specific). Recency-intent queries keep the composite-scale default 0.3."),
635
636
  collection: z.string().optional().describe("Filter to collection (single name or comma-separated)"),
636
637
  compact: z.boolean().optional().default(false).describe("Return compact results (id, path, title, score, snippet) instead of full content"),
637
638
  includeInternal: z.boolean().optional().default(false).describe("Include system-internal _clawmem docs (observations/deductions) — excluded by default"),
@@ -656,8 +657,17 @@ This is the recommended entry point for ALL memory queries.`,
656
657
 
657
658
  const coFn = (path: string) => store.getCoActivated(path);
658
659
  const enriched = enrichResults(store, results, query);
659
- const scored = applyCompositeScoring(enriched, query, coFn, mcpDirectWeightsOption())
660
- .filter(r => r.compositeScore >= (minScore || 0.3));
660
+ // v0.22.0 two-regime scoring (VSEARCH-RAW-PRIMARY-DESIGN.md R1–R4): non-recency
661
+ // queries rank by raw cosine — metadata (incl. pin) breaks exact score ties only,
662
+ // and minScore (if given) filters the raw score with NO default floor. Recency-intent
663
+ // queries keep the pre-v0.22.0 composite behavior including its 0.3 default.
664
+ const regime = selectScoringRegime(query);
665
+ const vsScoreBasis = regime === "raw" ? VECTOR_SCORE_BASIS : COMPOSITE_SCORE_BASIS;
666
+ const scored = regime === "raw"
667
+ ? rankRawPrimary(enriched, query, coFn).filter(r => minScore === undefined || r.compositeScore >= minScore)
668
+ // Recency branch preserves v0.21 semantics EXACTLY, including `||`: an explicit
669
+ // minScore of 0 still applies the 0.3 composite floor (R4 unchanged-recency contract).
670
+ : applyCompositeScoring(enriched, query, coFn).filter(r => r.compositeScore >= (minScore || 0.3));
661
671
 
662
672
  if (compact) {
663
673
  const items = scored.map(r => ({
@@ -666,7 +676,7 @@ This is the recommended entry point for ALL memory queries.`,
666
676
  snippet: (r.body || "").substring(0, 150), content_type: r.contentType, modified_at: r.modifiedAt,
667
677
  fragment: r.fragmentType ? { type: r.fragmentType, label: r.fragmentLabel } : undefined,
668
678
  }));
669
- return { content: [{ type: "text", text: `${formatSearchSummary(items.map(i => ({ ...i, file: i.path, compositeScore: i.score, context: null })), query)}${vsDegradedNote}` }], structuredContent: { results: items, ...vsDegraded } };
679
+ return { content: [{ type: "text", text: `${formatSearchSummary(items.map(i => ({ ...i, file: i.path, compositeScore: i.score, context: null })), query)}${vsDegradedNote}` }], structuredContent: { results: items, scoreBasis: vsScoreBasis, ...vsDegraded } };
670
680
  }
671
681
 
672
682
  const items: SearchResultItem[] = scored.map(r => {
@@ -685,7 +695,7 @@ This is the recommended entry point for ALL memory queries.`,
685
695
 
686
696
  return {
687
697
  content: [{ type: "text", text: `${formatSearchSummary(items, query)}${vsDegradedNote}` }],
688
- structuredContent: { results: items, ...vsDegraded },
698
+ structuredContent: { results: items, scoreBasis: vsScoreBasis, ...vsDegraded },
689
699
  };
690
700
  }
691
701
  );
@@ -729,12 +739,9 @@ This is the recommended entry point for ALL memory queries.`,
729
739
  : undefined;
730
740
  const excl = resolveExcludedCollections(includeInternal, collections);
731
741
  const initialFts = store.searchFTS(query, 20, undefined, collections, dateRange, excl);
732
- const topScore = initialFts.length > 0 ? Math.abs(initialFts[0]!.score) : 0;
733
- const secondScore = initialFts.length > 1 ? Math.abs(initialFts[1]!.score) : 0;
734
742
  // When intent is provided, disable strong-signal bypass — the obvious BM25
735
743
  // match may not be what the caller wants (e.g. "performance" with intent "web page load times")
736
- const hasStrongSignal = !intent && initialFts.length > 0
737
- && topScore >= 0.85 && (topScore - secondScore) >= 0.15;
744
+ const hasStrongSignal = !intent && hasStrongFtsSignal(initialFts);
738
745
 
739
746
  // Step 2: Query expansion (skipped if strong signal). Typed routing —
740
747
  // original → BOTH FTS + vector (2× RRF anchor), lex → FTS only, vec/hyde → vector only.
@@ -1083,9 +1090,11 @@ This is the recommended entry point for ALL memory queries.`,
1083
1090
 
1084
1091
  const top = candidates[0]!;
1085
1092
 
1086
- // Clear winner: high score OR significant gap to #2
1093
+ // Clear winner: high score OR significant gap to #2. The gap clause requires an
1094
+ // actual #2 to gap against — a lone weak candidate must not qualify (destructive
1095
+ // ops would otherwise auto-select a garbage single match).
1087
1096
  const gap = candidates.length > 1 ? top.score - candidates[1]!.score : 1.0;
1088
- const confident = top.score >= 0.7 || gap >= 0.2;
1097
+ const confident = top.score >= 0.7 || (candidates.length > 1 && gap >= 0.2);
1089
1098
 
1090
1099
  // For destructive ops (forget), require higher confidence
1091
1100
  if (destructive && !confident) {
@@ -1706,11 +1715,7 @@ This is the recommended entry point for ALL memory queries.`,
1706
1715
  const fusedRanked = reciprocalRankFusion([bm25Results.map(toRanked), vecResults.map(toRanked)], rrfWeights);
1707
1716
 
1708
1717
  // Map RRF results back to SearchResult with updated scores
1709
- const allSearchResults = [...bm25Results, ...vecResults];
1710
- const fused: SearchResult[] = fusedRanked.map(fr => {
1711
- const original = allSearchResults.find(r => r.filepath === fr.file);
1712
- return original ? { ...original, score: fr.score } : null;
1713
- }).filter((r): r is SearchResult => r !== null);
1718
+ const fused: SearchResult[] = attachRrfScores(fusedRanked, [...bm25Results, ...vecResults]);
1714
1719
 
1715
1720
  // Step 4: Graph expansion (if enabled and intent allows)
1716
1721
  let expanded = fused;
@@ -1904,10 +1909,11 @@ This is the recommended entry point for ALL memory queries.`,
1904
1909
  const vec = detGraph.results;
1905
1910
  if (detGraph.degraded && detGraph.degradedReason) degradedLegs.push({ leg: `clause${clauseIdx}:graph:vector`, reason: detGraph.degradedReason });
1906
1911
  const fused = reciprocalRankFusion([bm25.map(toRanked), vec.map(toRanked)], [1.0, 1.0]);
1907
- const searchMap = new Map([...bm25, ...vec].map(r => [r.filepath, r]));
1908
- results = fused
1909
- .map(fr => searchMap.get(fr.file))
1910
- .filter((r): r is SearchResult => r !== null);
1912
+ // Carry the RRF-fused score (parity with causal + intent_search): the graph
1913
+ // traversal below anchors on `score`, which must be RRF-scale, not raw single-channel.
1914
+ // vec-first: this site historically preferred the vector variant on duplicate
1915
+ // paths (its pre-helper Map construction let later entries overwrite) — preserved.
1916
+ results = attachRrfScores(fused, [...vec, ...bm25]);
1911
1917
 
1912
1918
  // Graph expansion for WHY/ENTITY
1913
1919
  if (intent.intent === 'WHY' || intent.intent === 'ENTITY') {
@@ -0,0 +1,79 @@
1
+ /**
2
+ * Centralized scoring-regime selection for the MCP direct vector routes (v0.22.0).
3
+ *
4
+ * Two regimes:
5
+ *
6
+ * "raw" — non-recency queries on the evidenced raw routes (`vsearch`,
7
+ * `memory_retrieve` semantic/discovery): results rank by RAW
8
+ * vector cosine DESC. Document metadata — including pin —
9
+ * participates ONLY inside groups of exactly-equal raw scores.
10
+ * The reported score IS the raw cosine (scoreBasis
11
+ * "vector-cosine"): specific to the embedding model that
12
+ * produced it, and not comparable to composite scores.
13
+ *
14
+ * "recency-composite" — hasRecencyIntent(query): the pre-v0.22.0 composite behavior,
15
+ * unchanged (RECENCY_WEIGHTS blend, multipliers, pin boost,
16
+ * contentType priority sort, composite-scale minScore default).
17
+ *
18
+ * Measured basis (BACKLOG Source 48): raw cosine ranked 16/19 judged targets #1
19
+ * (MRR 0.912) where the composite stack ranked 1/19 (MRR 0.307) and filtered 14/19
20
+ * below the old composite minScore — every metadata signal large enough to matter is
21
+ * larger than the 0.03–0.10 raw margins that separate right answers from wrong ones,
22
+ * so metadata may only ever break exact raw-score ties on these routes.
23
+ */
24
+ import {
25
+ applyCompositeScoring,
26
+ hasRecencyIntent,
27
+ type EnrichedResult,
28
+ type ScoredResult,
29
+ type CoActivationFn,
30
+ type CompositeScoringOptions,
31
+ } from "./memory.ts";
32
+
33
+ /** Reported score basis for raw-regime results: raw vector cosine. */
34
+ export const VECTOR_SCORE_BASIS = "vector-cosine" as const;
35
+ /** Reported score basis for composite-scored results. */
36
+ export const COMPOSITE_SCORE_BASIS = "composite" as const;
37
+
38
+ export type ScoringRegime = "raw" | "recency-composite";
39
+
40
+ /** One switch point for the raw-route regime decision (design R4). */
41
+ export function selectScoringRegime(query: string): ScoringRegime {
42
+ return hasRecencyIntent(query) ? "recency-composite" : "raw";
43
+ }
44
+
45
+ /**
46
+ * Raw-primary ranking (design R1/R2): raw score DESC; within a group of exactly-equal
47
+ * raw scores the deterministic tie order is pinned DESC, then legacy composite DESC,
48
+ * then displayPath ASC. `compositeScore` on the returned rows carries the RAW score —
49
+ * the reported score is the raw cosine.
50
+ *
51
+ * Callers gate on selectScoringRegime() first — this ranker is only meaningful for
52
+ * non-recency queries (the legacy composite computed here for tie keys therefore never
53
+ * takes the RECENCY_WEIGHTS branch). Inputs are document-unique on these routes
54
+ * (searchVecDetailed hydrates one row per document).
55
+ *
56
+ * `options.now` flows to the tie-key composite so frozen-clock evaluation (the v0.22.0
57
+ * acceptance bundle) is bit-reproducible.
58
+ */
59
+ export function rankRawPrimary(
60
+ results: EnrichedResult[],
61
+ query: string,
62
+ coActivationFn?: CoActivationFn,
63
+ options?: CompositeScoringOptions
64
+ ): ScoredResult[] {
65
+ const legacy = applyCompositeScoring(results, query, coActivationFn, options);
66
+ const legacyByPath = new Map(legacy.map(s => [s.filepath, s]));
67
+ const ranked = results.map(r => {
68
+ const l = legacyByPath.get(r.filepath) ?? ({ ...r, compositeScore: r.score, recencyScore: 0 } as ScoredResult);
69
+ return { row: { ...l, compositeScore: r.score } as ScoredResult, legacyComposite: l.compositeScore };
70
+ });
71
+ ranked.sort((a, b) => {
72
+ if (b.row.compositeScore !== a.row.compositeScore) return b.row.compositeScore - a.row.compositeScore;
73
+ const pinDelta = (b.row.pinned ? 1 : 0) - (a.row.pinned ? 1 : 0);
74
+ if (pinDelta !== 0) return pinDelta;
75
+ if (b.legacyComposite !== a.legacyComposite) return b.legacyComposite - a.legacyComposite;
76
+ return a.row.displayPath < b.row.displayPath ? -1 : a.row.displayPath > b.row.displayPath ? 1 : 0;
77
+ });
78
+ return ranked.map(r => r.row);
79
+ }
@@ -201,3 +201,35 @@ export function toRanked(r: SearchResult): RankedResult {
201
201
  score: r.score,
202
202
  };
203
203
  }
204
+
205
+ /**
206
+ * Map RRF-fused results back to their originating SearchResults, carrying the FUSED
207
+ * score. Downstream consumers — graph-traversal anchors, composite enrichment — must
208
+ * see the fusion's relevance scale, not a raw single-channel score (S49.1 U2: the
209
+ * query_plan graph clause used to drop the fused score, anchoring traversal on raw
210
+ * FTS values). First occurrence wins when a doc appears in multiple channels
211
+ * (matching the previous `find`-based lookups); fused entries with no originating
212
+ * SearchResult are dropped.
213
+ */
214
+ export function attachRrfScores(fused: RankedResult[], originals: SearchResult[]): SearchResult[] {
215
+ const byPath = new Map<string, SearchResult>();
216
+ for (const r of originals) {
217
+ if (!byPath.has(r.filepath)) byPath.set(r.filepath, r);
218
+ }
219
+ return fused
220
+ .map(fr => { const orig = byPath.get(fr.file); return orig ? { ...orig, score: fr.score } : null; })
221
+ .filter((r): r is SearchResult => r !== null);
222
+ }
223
+
224
+ /**
225
+ * BM25 strong-signal check on ftsScoreFromBm25-scale results: the top hit is strong
226
+ * (≥ 0.85 ⇔ |bm25| ≥ 5.67) AND clearly separated from #2 (gap ≥ 0.15; a missing #2
227
+ * counts as 0, so a lone WEAK hit does not qualify). Callers use this to skip LLM
228
+ * query expansion when the keyword answer is unambiguous.
229
+ */
230
+ export function hasStrongFtsSignal(results: SearchResult[]): boolean {
231
+ if (results.length === 0) return false;
232
+ const top = results[0]!.score;
233
+ const second = results.length > 1 ? results[1]!.score : 0;
234
+ return top >= 0.85 && (top - second) >= 0.15;
235
+ }
package/src/server.ts CHANGED
@@ -12,7 +12,7 @@
12
12
 
13
13
  import type { Server } from "bun";
14
14
  import type { Store, SearchResult, TimelineResult } from "./store.ts";
15
- import { enrichResults, reciprocalRankFusion, toRanked } from "./search-utils.ts";
15
+ import { enrichResults, reciprocalRankFusion, toRanked, attachRrfScores } from "./search-utils.ts";
16
16
  import { applyCompositeScoring, hasRecencyIntent, type EnrichedResult } from "./memory.ts";
17
17
  import { applyMMRDiversity } from "./mmr.ts";
18
18
  import { listCollections } from "./collections.ts";
@@ -617,11 +617,7 @@ async function handleRetrieve(req: Request, _url: URL, store: Store): Promise<Re
617
617
  } catch (e) { rethrowIfFatalVectorError(e); /* vector unavailable */ }
618
618
  const weights = intent.intent === "WHEN" ? [1.5, 1.0] : [1.0, 1.5];
619
619
  const fused = reciprocalRankFusion([bm25.map(toRanked), vec.map(toRanked)], weights);
620
- const allResults = [...bm25, ...vec];
621
- results = fused.map(fr => {
622
- const orig = allResults.find(r => r.filepath === fr.file);
623
- return orig ? { ...orig, score: fr.score } : null;
624
- }).filter((r): r is SearchResult => r !== null);
620
+ results = attachRrfScores(fused, [...bm25, ...vec]);
625
621
  } else if (mode === "keyword") {
626
622
  results = store.searchFTS(query, limit * 2, undefined, collections);
627
623
  } else if (mode === "semantic") {
package/src/store.ts CHANGED
@@ -3506,6 +3506,20 @@ function buildFTS5Query(query: string): string | null {
3506
3506
  return terms.map(t => `"${t}"*`).join(' AND ');
3507
3507
  }
3508
3508
 
3509
+ /**
3510
+ * Convert an FTS5 bm25() value into a stable [0,1) relevance score where higher is better.
3511
+ *
3512
+ * FTS5's bm25() is negative-is-better: it returns -1 × the BM25 score, so it is ≤ 0 for
3513
+ * every match. The transform is per-row and monotonic in match strength (|bm25|/(1+|bm25|))
3514
+ * with no per-query normalization, so cross-query comparisons, minScore filters, and the
3515
+ * strong-signal bypass all stay meaningful. A hypothetical positive input clamps to 0
3516
+ * rather than inverting the ordering.
3517
+ */
3518
+ export function ftsScoreFromBm25(bm25Score: number): number {
3519
+ const m = Math.max(0, -bm25Score);
3520
+ return m / (1 + m);
3521
+ }
3522
+
3509
3523
  export function searchFTS(db: Database, query: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, excludeCollections?: string[]): SearchResult[] {
3510
3524
  const ftsQuery = buildFTS5Query(query);
3511
3525
  if (!ftsQuery) return [];
@@ -3558,9 +3572,7 @@ export function searchFTS(db: Database, query: string, limit: number = 20, colle
3558
3572
  const rows = db.prepare(sql).all(...params) as { filepath: string; display_path: string; title: string; body: string; hash: string; modified_at: string; bm25_score: number }[];
3559
3573
  return rows.map(row => {
3560
3574
  const collectionName = row.filepath.split('//')[1]?.split('/')[0] || "";
3561
- // Convert bm25 (lower is better) into a stable (0..1] score where higher is better.
3562
- // Avoid per-query normalization so "strong signal" heuristics can work.
3563
- const score = 1 / (1 + Math.max(0, row.bm25_score));
3575
+ const score = ftsScoreFromBm25(row.bm25_score);
3564
3576
  return {
3565
3577
  filepath: row.filepath,
3566
3578
  displayPath: row.display_path,