clawmem 0.20.2 → 0.22.0

package/AGENTS.md

@@ -59,7 +59,7 @@ clawmem doctor                              # full health check (clawmem status
 | `feedback-loop` | Stop | tracks referenced notes → confidence boosts, co-activations, utility signals |
 
 **Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
-**Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
+**Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3. Note the MCP retrieval tools themselves exclude `_clawmem` by default since v0.21.0 — pass `includeInternal: true` when system-internal memory (observations/handoffs/deductions) is the target.
 **Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) + an activation floor; only `deep` adds query expansion + reranking to the hook path. → [docs/concepts/hooks-vs-mcp.md](docs/concepts/hooks-vs-mcp.md).
 
 ### Tier 3 — agent-initiated (one targeted MCP call)
@@ -105,7 +105,7 @@ All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculat
 | `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why". |
 | `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
 | `profile` | User profile (static facts + dynamic context). |
-| `memory_pin` | +0.3 composite boost. PROACTIVELY for constraints, architecture decisions, corrections. |
+| `memory_pin` | Lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on composite surfaces; exact-tie precedence on raw vector routes). PROACTIVELY for constraints, architecture decisions, corrections. |
 | `memory_snooze` | PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
 | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
 | `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
@@ -135,14 +135,14 @@ The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector v
 
 ## Composite scoring
 
-Applied automatically to all search results.
+Applied on the composite surfaces: hooks, `query`, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. **v0.22.0: MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine instead** (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default); recency-intent queries keep composite everywhere.
 
 ```
 compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
 ```
 
-- `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× … 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive**.
-- **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/context-surfacing keep the 0.50/0.25/0.25 default.
+- `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× … 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive on composite surfaces** (raw vector routes: pin = exact-tie precedence only).
+- **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`, `memory_retrieve`'s composite modes, and context-surfacing keep the 0.50/0.25/0.25 default. (`vsearch` + `memory_retrieve` semantic/discovery use RAW cosine for non-recency queries as of v0.22.0 — no composite weights at all.)
 - **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
 - **Half-lives:** decision/deductive/preference/hub/antipattern = ∞ · project 120d · research 90d · problem/milestone/note 60d · conversation/progress 45d · handoff 30d (extend up to 3× for frequently-accessed).
 
@@ -162,7 +162,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
 
 ## Memory lifecycle (pin / snooze / forget — manual tools)
 
-- **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
+- **`memory_pin`** (lifecycle retention + priority among relevance-equivalent results; +0.3 boost on composite surfaces, exact-tie precedence on raw vector routes) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
 - **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
 - **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
 - **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.

package/README.md

@@ -531,6 +531,8 @@ clawmem status                                  Quick index status
 
 Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
 
+**Internal-collection visibility (v0.21.0):** the retrieval tools `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` exclude the system-internal `_clawmem` collection by default — pass `includeInternal: true` (or name `_clawmem` in an explicit `collection` filter where the tool has one) to include it. `find_similar` auto-includes internal neighbors when the reference document is itself internal; `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are unfiltered by design. Full contract (including the `degraded` under-fill markers): [docs/reference/mcp-tools.md](docs/reference/mcp-tools.md).
+
 | Tool | Description |
 |---|---|
 | `__IMPORTANT` | Workflow guide: prefer `memory_retrieve` → match query type to tool → `multi_get` for full content |
@@ -541,7 +543,7 @@ Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
 |---|---|
 | `memory_retrieve` | **Preferred entry point.** Auto-classifies query and routes to optimal backend (query, intent_search, session_log, find_similar, or query_plan). Use instead of manually choosing a search tool. |
 | `search` | BM25 keyword search — for exact terms, config names, error codes, filenames. Composite scoring + co-activation boost + compact mode. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
-| `vsearch` | Vector semantic search — for conceptual/fuzzy matching when exact keywords are unknown. Composite scoring + co-activation boost + compact mode. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
+| `vsearch` | Vector semantic search — for conceptual/fuzzy matching when exact keywords are unknown. **v0.22.0: non-recency queries rank by RAW cosine (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default)**; recency-intent queries keep composite. Collection filter supports comma-separated values. Prefer `memory_retrieve` for auto-routing. |
 | `query` | Full hybrid pipeline (BM25 + vector + rerank) — general-purpose when query type is unclear. WRONG for "why" questions (use `intent_search`) or cross-session queries (use `session_log`). Prefer `memory_retrieve` for auto-routing. Intent hint, strong-signal bypass, chunk dedup, candidateLimit, MMR diversity, compact mode. |
 | `get` | Retrieve single document by path or docid |
 | `multi_get` | Retrieve multiple docs by glob or comma-separated list |
@@ -593,7 +595,7 @@ Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
 | Tool | Description |
 |---|---|
 | `memory_forget` | Search → deactivate closest match (with audit trail) |
-| `memory_pin` | Pin a memory for +0.3 composite boost. USE PROACTIVELY when: user states a persistent constraint, makes an architecture decision, or corrects a misconception. Don't wait for curator — pin critical decisions immediately. |
+| `memory_pin` | Pin a memory: lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on hook/`query`/`search` surfaces; exact-tie precedence on the raw vector routes). USE PROACTIVELY when: user states a persistent constraint, makes an architecture decision, or corrects a misconception. Don't wait for curator — pin critical decisions immediately. |
 | `memory_snooze` | Temporarily hide a memory from context surfacing until a date. USE PROACTIVELY when `<vault-context>` repeatedly surfaces irrelevant content — snooze for 30 days instead of ignoring it. |
 | `status` | Index health with content type distribution |
 | `reindex` | Trigger vault re-scan |
@@ -693,7 +695,7 @@ Content types are inferred from frontmatter or file path patterns. Half-lives ex
 
 **Frequency boost:** Documents with higher revision counts or duplicate counts get a durability signal: `freqSignal = (revisions - 1) × 2 + (duplicates - 1)`, `freqBoost = min(0.10, log1p(freqSignal) × 0.03)`. Revision count (content evolution) is weighted 2× vs duplicate count (ingest repetition). Capped at 10%.
 
-**Pin boost:** Pinned documents get +0.3 additive boost (capped at 1.0). Use `memory_pin` to pin critical memories.
+**Pin boost:** Pin = lifecycle retention + priority among relevance-equivalent results. On composite surfaces pinned documents get a +0.3 additive boost (capped at 1.0); on the raw vector routes (v0.22.0) pin breaks exact raw-score ties only. Use `memory_pin` to pin critical memories.
 
 **Snooze:** Snoozed documents are filtered out of context surfacing until their snooze date. Use `memory_snooze` for temporary suppression.
 
@@ -789,6 +791,7 @@ Notes referenced by the agent during a session get boosted (`access_count++`). U
 | `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server URL. Without it, falls to `node-llama-cpp` (if allowed). |
 | `CLAWMEM_RERANK_API_KEY` | (none) | Bearer token for an authenticated remote reranker endpoint. Independent of the embed and LLM keys. |
 | `CLAWMEM_NO_LOCAL_MODELS` | `false` | Block `node-llama-cpp` from auto-downloading GGUF models. Set `true` for remote-only setups where you want fail-fast on unreachable endpoints. |
+| `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | (superseded) | **No effect since v0.22.0** — the direct-pipeline eval this knob was gated on measured tuned weights at 1/19 hit@1; the direct vector routes rank by raw cosine instead. Still parsed for backward compatibility; setting it logs a once-per-process warning. |
 | `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 consolidation merge-safety threshold when candidate and existing anchors align. Merges above this normalized 3-gram cosine score are allowed. |
 | `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety threshold — fallback when anchor sets are ambiguous. |
 | `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration before enabling the gate. |

package/SKILL.md

@@ -35,7 +35,7 @@ Hooks handle ~90% of retrieval at zero agent effort.
 
 **Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
 
-**Hook blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, and cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
+**Hook blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, and cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3. Note the MCP retrieval tools themselves exclude `_clawmem` by default since v0.21.0 — pass `includeInternal: true` when system-internal memory (observations/handoffs/deductions) is the target.
 
 **Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) and an activation floor. Only `deep` adds query expansion + reranking to the hook path. Profile and the hook `timeout` are set in `~/.claude/settings.json` — see *Operational gotchas* for timeout tuning.
 
@@ -92,7 +92,7 @@ All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculat
 | `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why" (use `intent_search`). |
 | `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
 | `profile` | User profile (static facts + dynamic context). |
-| `memory_pin` | +0.3 composite boost. Use PROACTIVELY for constraints, architecture decisions, corrections. |
+| `memory_pin` | Lifecycle retention + priority among relevance-equivalent results (+0.3 composite boost on composite surfaces; exact-tie precedence on raw vector routes). Use PROACTIVELY for constraints, architecture decisions, corrections. |
 | `memory_snooze` | Use PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
 | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
 | `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
@@ -189,7 +189,7 @@ Query -> intent classification (WHY/WHEN/ENTITY/WHAT)
 
 ## Composite scoring (how ranking works)
 
-Applied automatically to all search results.
+Applied on the composite surfaces: hooks, `query`, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. **v0.22.0: MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine instead** (`scoreBasis: "vector-cosine"`; metadata breaks exact ties only; `minScore` filters raw with no default); recency-intent queries keep composite everywhere.
 
 ```
 compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
@@ -198,8 +198,8 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
 - `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× penalty … 1.3× boost).
 - `coActivationBoost = 1 + min(coCount/10, 0.15)` (docs surfaced together get up to +15%).
 - Length normalization penalizes verbose entries (floor 30%); frequency boost capped at +10%.
-- **Pinned docs: +0.3 additive** (capped at 1.0).
-- **`query` tool (v0.13.0+):** non-recency queries use retrieval-tuned **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/`context-surfacing` keep the 0.50/0.25/0.25 default.
+- **Pinned docs: +0.3 additive on composite surfaces** (capped at 1.0); on the raw vector routes pin = exact-tie precedence only.
+- **`query` tool (v0.13.0+):** non-recency queries use retrieval-tuned **0.70·search + 0.15·recency + 0.15·confidence**. `search`, `memory_retrieve`'s composite modes, and `context-surfacing` keep the 0.50/0.25/0.25 default. (`vsearch` + `memory_retrieve` semantic/discovery use RAW cosine for non-recency queries as of v0.22.0 — no composite weights at all.)
 - **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
 
 **Content-type half-lives:** decision / deductive / preference / hub / antipattern = ∞ (never decay) · project 120d · research 90d · problem / milestone / note 60d · conversation / progress 45d · handoff 30d. Half-lives extend up to 3× for frequently-accessed memories. Attention decay: non-durable types (handoff, progress, conversation, note, project) lose 5% confidence/week without access; decision / deductive / preference / hub / research / antipattern are exempt.
@@ -210,7 +210,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
 
 ## Memory lifecycle (pin / snooze / forget — manual tools)
 
-- **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist across sessions. Do NOT pin routine/session-specific items.
+- **`memory_pin`** (lifecycle retention + priority among relevance-equivalent results; +0.3 boost on composite surfaces, exact-tie precedence on raw vector routes) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist across sessions. Do NOT pin routine/session-specific items.
 - **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
 - **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
 - **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.

package/docs/concepts/architecture.md

@@ -56,7 +56,7 @@ Each indexed file becomes a document with:
 | `content_type` | Auto-detected: decision, deductive, preference, note, handoff, conversation, progress, research, hub, antipattern, project, milestone, problem |
 | `quality_score` | 0.0-1.0 based on length, structure, headings, lists, decision keywords, frontmatter |
 | `confidence` | Starts at 0.5, adjusted by contradiction detection and feedback |
-| `pinned` | Manual boost flag (+0.3 composite score) |
+| `pinned` | Lifecycle retention + priority among relevance-equivalent results (+0.3 on composite surfaces; exact-tie precedence on the raw vector routes) |
 | `snoozed_until` | Temporarily hidden from context surfacing |
 
 ## Fragments

package/docs/concepts/composite-scoring.md

@@ -1,6 +1,6 @@
 # Composite Scoring
 
-Every search result in ClawMem is scored using a composite formula that blends multiple signals beyond raw search relevance.
+Composite scoring blends multiple signals beyond raw search relevance. **It applies to the hook pipeline (context surfacing), the `query` hybrid pipeline, `search`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes.** Since v0.22.0 the direct vector routes — MCP `vsearch` and `memory_retrieve` semantic/discovery — do NOT use it for non-recency queries: they rank by raw vector cosine, with metadata (including pin) breaking exact score ties only, and report `scoreBasis: "vector-cosine"`. That split is measured, not aesthetic: on a judged set against the live vault, raw cosine ranked 16/19 targets #1 (MRR 0.912) while this composite ranked 1/19 (MRR 0.307) and pushed 14/19 correct answers below the old minScore floor — in the compressed-high similarity band of modern embedding models, every multiplier below is larger than the raw margins separating right answers from wrong ones. Recency-intent queries keep composite (the Recency column) on every route that had it.
 
 ## Formula
 
@@ -19,7 +19,9 @@ compositeScore = (w_search * searchScore + w_recency * recencyScore + w_confiden
 
 Recency intent is detected automatically when queries contain "latest", "recent", "last session", etc. — and **takes precedence over the `query`-tool weights**: a recency-phrased `query` call still uses the Recency-intent column.
 
-The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. All other tools (`search`, `vsearch`, `memory_retrieve`, and the context-surfacing hook) use the **Normal** column.
+The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. `search`, the composite `memory_retrieve` modes, and the context-surfacing hook **keep the Normal column**. The v0.21.0 `retrieval.mcp_direct_tuned_weights` knob is **superseded as of v0.22.0 and has no effect** — the direct-pipeline eval it was gated on measured tuned weights at 1/19 hit@1, and the direct vector routes moved to raw ordering instead. The key is still parsed (a once-per-process warning is logged if set) so existing configs don't break.
+
+Threshold note: EOS-anchored last-token embedding models (e.g. zembed-1 served correctly) produce a **compressed-high similarity band** — unrelated pairs sit near ~0.4 rather than ~0.2. Relative ordering is what matters; treat absolute `minScore` cutoffs as model-dependent rather than universal.
 
 ## Signal breakdown
 
@@ -90,13 +92,13 @@ coActivationBoost = 1 + min(coCount / 10, 0.15)
 
 Documents frequently surfaced together in the same session get up to 15% boost.
 
-**Where co-activation is applied depends on the caller.** MCP tools (`query`, `search`, `vsearch`) pass a co-activation function into `applyCompositeScoring()`, so the boost is part of the composite score used for ranking and `minScore` filtering. The context-surfacing hook does NOT pass co-activation into composite scoring — instead it applies a separate spreading-activation step *after* adaptive threshold filtering. This means the hook's threshold decisions are based on scores without co-activation, and co-activation only boosts results that already passed the threshold. This is intentional: it prevents relationship boosts from rescuing otherwise-weak results into the surfaced set.
+**Where co-activation is applied depends on the caller.** The composite MCP surfaces (`query`, `search`, and `vsearch` on recency-intent queries only) pass a co-activation function into `applyCompositeScoring()`, so the boost is part of the composite score used for ranking and — on `vsearch` and `search`, the tools that expose `minScore` — for `minScore` filtering. On `vsearch`'s non-recency (raw) regime, co-activation contributes only to the exact-tie key, never to ranking or filtering. The context-surfacing hook does NOT pass co-activation into composite scoring — instead it applies a separate spreading-activation step *after* adaptive threshold filtering. This means the hook's threshold decisions are based on scores without co-activation, and co-activation only boosts results that already passed the threshold. This is intentional: it prevents relationship boosts from rescuing otherwise-weak results into the surfaced set.
 
 ## Additional modifiers
 
 ### Pin boost
 
-Pinned documents receive +0.3 additive boost (capped at 1.0 total).
+On composite surfaces, pinned documents receive a +0.3 additive boost (capped at 1.0 total). Pin's contract is **lifecycle retention + prioritization among relevance-equivalent results** — on the raw vector routes (v0.22.0) it therefore breaks exact raw-score ties only and never lifts a document over a more relevant one.
 
 ### Length normalization
 

package/docs/guides/inference-services.md

@@ -60,9 +60,18 @@ ZeroEntropy's distillation-paired stack — best retrieval quality, total ~16 GB
 | Reranker | 8090 | [zerank-2 seq-cls sidecar](../../extras/rerankers/zerank-2-seq/) (transformers, bf16) | ~9 GB | SOTA reranker — **not** a GGUF |
 
 ```bash
-# Embedding (zembed-1) — -ub MUST equal -b for non-causal attention
+# Embedding (zembed-1) — -ub MUST equal -b for non-causal attention.
+# --pooling last: zembed-1 is a last-token model; declare it explicitly.
+# --override-kv ...add_eos_token: last-token models read the embedding at their EOS
+# anchor. If the GGUF conversion lost add_eos_token, the server never appends the
+# terminator and similarity collapses to last-token identity (see troubleshooting →
+# "Vector search returns weak or irrelevant results"). The override restores it and
+# is a no-op when the metadata is already correct. Do NOT add these two flags to
+# mean-pooling models like the default EmbeddingGemma.
 llama-server -m zembed-1-Q4_K_M.gguf \
-  --embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
+  --embeddings --pooling last \
+  --override-kv tokenizer.ggml.add_eos_token=bool:true \
+  --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
 
 # Reranker (zerank-2) — seq-cls SIDECAR (transformers, bf16), NOT a llama-server GGUF:
 cd extras/rerankers/zerank-2-seq

package/docs/guides/upgrading.md

@@ -1,6 +1,6 @@
 # Upgrading ClawMem
 
-Guide for upgrading between released versions. Current: **v0.10.4**.
+Guide for upgrading between released versions. Current: **v0.22.0**.
 
 ClawMem upgrades are designed to be drop-in: pull the new version, restart any long-lived processes, and the SQLite schema auto-migrates on first open. This guide documents per-version specifics for upgrades that have additional considerations beyond the quick path below.
 
@@ -59,6 +59,36 @@ docker compose up -d reranker                      # /v1/rerank on :8090
 
 ---
 
+## v0.22.0: raw-cosine ranking on the direct vector routes
+
+No migration command, no schema change, no re-embed. Restart long-lived processes per the quick path.
+
+**Behavior change — MCP `vsearch` and `memory_retrieve` semantic/discovery rank non-recency queries by RAW cosine.** Reported scores on those routes are raw cosine (`scoreBasis: "vector-cosine"`), not composite values — expect a different numeric scale (unrelated results sit near ~0.4–0.55 on compressed-band embedding models, not near 0). `vsearch`'s `minScore` filters that raw scale and no longer defaults to 0.3 — omitted means no filter. If a workflow passed `minScore` tuned to composite values, re-tune it to your embedding model's band or omit it. Recency-phrased queries ("latest…", "recently…") behave exactly as before on every route.
+
+**Pinned documents** no longer float above more relevant results on those two routes — pin now means lifecycle retention plus winning exact-score ties. Hooks, `query`, and `search` keep the +0.3 composite pin boost.
+
+**`retrieval.mcp_direct_tuned_weights` / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` no longer has any effect** (superseded by its own gating eval). Configs that set it keep working; a once-per-process warning is logged — remove the key at your convenience.
+
+---
+
+## v0.21.0: MCP internal-collection exclusion + embedding-geometry canary
+
+No migration command required. Two things to know:
+
+**Behavior change — `_clawmem` excluded from MCP retrieval by default.** `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` no longer return the system-internal `_clawmem` collection (observations/deductions/handoffs) unless asked. If a workflow depended on those appearing in MCP results, pass `includeInternal: true` or name `_clawmem` in an explicit `collection` filter. `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are unfiltered by design, and hooks already filtered internal docs — `<vault-context>` behavior is unchanged. Details: [docs/reference/mcp-tools.md](../reference/mcp-tools.md).
+
+**What auto-applies on first open** (additive, idempotent — same contract as prior releases): the `embed_canary` and `vault_flags` tables, and an `embed_input_fp` column on `content_vectors`.
+
+**No re-embed required.** Existing vectors stay valid. Pre-0.21.0 rows carry no input fingerprint, so `clawmem doctor`'s new sampled vector validation checks them structurally and flags title provenance as unavailable ("legacy") until each document's next natural re-embed — informational, not an error.
+
+**The canary baseline seeds itself.** The first `clawmem embed` run after upgrade (including a timer-fired one) probes the embedding server with a pair-separation battery and persists a first-healthy baseline; subsequent runs alert relative to it, and a broken-geometry server now aborts `embed --force` BEFORE anything is cleared. `--force-geometry` (proceed despite a failed probe; the vault is flagged tainted until a verified rebuild) and `--force --recalibrate-canary` (replace the baseline after a deliberate model/server change) are operator overrides, not upgrade steps.
+
+**Opt-in knob:** `retrieval.mcp_direct_tuned_weights` (config) / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` (env), default `false` — scored the MCP direct tools' non-recency queries with the retrieval-tuned `query`-tool weights. *(Superseded in v0.22.0 — the knob no longer has any effect; see the v0.22.0 section above.)*
+
+Upgrades from v0.13 → v0.20 shipped no steps beyond the [quick path](#quick-path) — schema changes auto-apply; see [RELEASE_NOTES.md](../../RELEASE_NOTES.md) for what each version changed.
+
+---
+
 ## v0.12.0: query reranking blend (no action required)
 
 v0.12.0 changes the `query` tool's rerank/RRF blend so the cross-encoder reranker can promote the best document to the top — the previous blend left RRF #1 mathematically immovable. It is a pure ranking-quality change: **no migration, no schema change, no config change.** It applies automatically on upgrade. Hooks and the per-session MCP stdio server pick it up on their next invocation; restart any long-lived `query` host — `clawmem serve` and persistent MCP/daemon processes — to pick up the improved ordering. The default reranker (`qwen3-reranker-0.6B`) is unchanged, and the blend improvement applies whatever reranker `:8090` serves. (`intent_search` and the context-surfacing hook keep their existing blends.)
@@ -145,7 +175,7 @@ Hooks and the stdio MCP server pick up the new binary automatically on their nex
 - **`plugins.slots.memory: "clawmem"` replaces `plugins.slots.contextEngine: "clawmem"`.** On the new pure-memory plugin, the exclusive slot is `memory`. The `setup openclaw` next-steps output tells you to run `openclaw plugins enable clawmem`, which sets the slot and disables competing memory plugins (`memory-core`, `memory-lancedb`) in a single command. You do NOT need to run the older `openclaw config set plugins.slots.contextEngine clawmem` pattern on v0.10.0.
 - **`src/openclaw/package.json` is now the plugin's discovery manifest.** OpenClaw v2026.4.11's `discoverInDirectory` reads `package.json` for the `openclaw.extensions` field and uses that to decide whether a directory under `~/.openclaw/extensions/` is a valid plugin. The older `openclaw.plugin.json` manifest is still shipped and parsed at runtime, but it is not sufficient to pass discovery on v2026.4.11+ without the `package.json` companion file. v0.10.0 adds the `package.json` to the plugin source tree, and `clawmem setup openclaw` verifies it is present before copying.
 - **`clawmem setup openclaw` defaults to recursive copy instead of symlink.** OpenClaw v2026.4.11 walks `~/.openclaw/extensions/` with `readdirSync({ withFileTypes: true })` and uses `dirent.isDirectory()` to descend into candidate plugin directories. Symlinks to directories report `isDirectory() === false` on that API shape, so a symlinked plugin is silently skipped during discovery. v0.10.0's `cmdSetupOpenClaw` therefore copies the plugin source into `~/.openclaw/extensions/clawmem/` with `cpSync(..., { recursive: true, dereference: true })`. A `--link` opt-in flag preserves the old symlink behavior for local development and for older OpenClaw versions, with a warning that v2026.4.11+ discovery will skip the symlink. Setup is idempotent: any existing plugin directory or stale symlink is removed before the new copy is written.
-- **Multi-user ownership check (OpenClaw v2026.4.11+).** If the gateway runs as a dedicated system user (e.g. `openclaw`) and you run `clawmem setup openclaw` as a different user (e.g. `sciros`), the copied plugin directory is owned by the installer user, but OpenClaw's ownership check rejects it with `suspicious ownership (uid=1001, expected uid=997 or root)`. This is a security feature that prevents a privileged gateway process from loading code a less-privileged user dropped into its extensions directory. Fix: `sudo chown -R <gateway-user>:<gateway-group> ~/.openclaw/extensions/clawmem`. Single-user installs where you ARE the gateway user are not affected — your own user owns the copy, and the ownership check passes.
+- **Multi-user ownership check (OpenClaw v2026.4.11+).** If the gateway runs as a dedicated system user (e.g. `openclaw`) and you run `clawmem setup openclaw` as a different user (e.g. `alice`), the copied plugin directory is owned by the installer user, but OpenClaw's ownership check rejects it with `suspicious ownership (uid=1001, expected uid=997 or root)`. This is a security feature that prevents a privileged gateway process from loading code a less-privileged user dropped into its extensions directory. Fix: `sudo chown -R <gateway-user>:<gateway-group> ~/.openclaw/extensions/clawmem`. Single-user installs where you ARE the gateway user are not affected — your own user owns the copy, and the ownership check passes.
 
 ### Rollback
 
@@ -191,7 +221,7 @@ Hooks + MCP stdio pick up new code automatically on next invocation — no resta
 
 When the user's prompt mentions entities already known to the vault (via `entity_nodes`), `context-surfacing` now appends a token-bounded `<vault-facts>` block of raw SPO triple lines to `<vault-context>`, alongside the existing `<facts>` / `<relationships>` blocks. This feeds the model current-state knowledge about entities the user is talking about, without requiring the agent to call `kg_query` explicitly.
 
-- **Three-path entity seeding** — canonical-ID regex (e.g. `default:project:clawmem`) → proper-noun extraction via `resolveEntityTypeExact` → longer-first n-gram scan (3-gram > 2-gram > 1-gram) for lowercase/hyphenated vocabulary like `forge-stack`, `oauth2`, `vm 202`. All three paths run prompt-only — entity seeds NEVER come from surfaced doc bodies, so topic-boosted off-topic docs cannot pollute the facts block.
+- **Three-path entity seeding** — canonical-ID regex (e.g. `default:project:clawmem`) → proper-noun extraction via `resolveEntityTypeExact` → longer-first n-gram scan (3-gram > 2-gram > 1-gram) for lowercase/hyphenated vocabulary like `side-project`, `oauth2`, `vm 12`. All three paths run prompt-only — entity seeds NEVER come from surfaced doc bodies, so topic-boosted off-topic docs cannot pollute the facts block.
 - **Profile-gated token sub-budget** — `factsTokens=0` on `speed` disables the stage entirely. `balanced` uses 200 tokens, `deep` uses 250. The sub-budget is dedicated — `<vault-facts>` cannot steal budget from `<facts>` or `<relationships>`.
 - **Truncation** — at the triple boundary, never mid-triple, never emits an empty block.
 - **Fail-open** — empty entity set → skip. Budget too small → drop block. Per-entity DB error → skip that entity. Any exception in the stage → return baseline `vault-context` unchanged.

package/docs/introduction.md

@@ -6,7 +6,7 @@ ClawMem is an open-source memory engine for Claude Code and AI agents. It runs o
 
 - **Indexes** markdown documents into a local SQLite vault with full-text search (BM25) and vector embeddings
 - **Retrieves** relevant context automatically via hooks or on-demand via MCP tools / REST API
-- **Scores** results using composite scoring (relevance, recency, confidence, quality, co-activation)
+- **Scores** results per route: raw vector cosine on the direct vector tools (v0.22.0), composite scoring (relevance, recency, confidence, quality, co-activation) on the hook and hybrid pipelines
 - **Tracks** decisions, handoffs, and session history across conversations
 - **Traverses** causal and semantic graphs to answer "why" and "what led to" questions
 

package/docs/quickstart.md

@@ -70,7 +70,7 @@ llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
   --reranking --port 8090 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 512
 ```
 
-> **SOTA upgrade (16GB+ GPU):** Replace embedding with zembed-1-Q4_K_M (2560d, `-b 2048 -ub 2048`). For the reranker, use the **zerank-2 seq-cls sidecar** (transformers, bf16) at [`extras/rerankers/zerank-2-seq/`](../extras/rerankers/zerank-2-seq/) — the `zerank-2-Q4_K_M` GGUF is deprecated (llama.cpp drops its score head). See [the inference services guide](guides/inference-services.md). **CC-BY-NC-4.0** — non-commercial only.
+> **SOTA upgrade (16GB+ GPU):** Replace embedding with zembed-1-Q4_K_M (2560d, `-b 2048 -ub 2048`, **plus `--pooling last --override-kv tokenizer.ggml.add_eos_token=bool:true`** — last-token models need their EOS anchor; the exact launch line is in [the inference services guide](guides/inference-services.md)). For the reranker, use the **zerank-2 seq-cls sidecar** (transformers, bf16) at [`extras/rerankers/zerank-2-seq/`](../extras/rerankers/zerank-2-seq/) — the `zerank-2-Q4_K_M` GGUF is deprecated (llama.cpp drops its score head). **CC-BY-NC-4.0** — non-commercial only.
 
 See [GPU services guide](guides/systemd-services.md) for systemd setup and remote GPU configuration.
 

package/docs/reference/cli.md

@@ -7,7 +7,7 @@ Complete command reference for the ClawMem memory engine. Always use the `bin/cl
 ```bash
 clawmem init                    # Initialize vault (creates SQLite DB)
 clawmem status                  # Quick index status
-clawmem doctor                  # Full health check (GPU connectivity, index integrity)
+clawmem doctor                  # Full health check (GPU connectivity, index integrity, embedding-geometry canary, sampled vector validation)
 ```
 
 ## Collection management
@@ -30,8 +30,10 @@ clawmem mine <dir> --dry-run                   # Preview without importing
 clawmem mine <dir> --synthesize                # v0.7.2: import + post-import LLM fact extraction
 clawmem mine <dir> --synthesize --synthesis-max-docs 50   # Cap synthesis to first 50 conversations (default 20)
 clawmem reindex                                # Force re-scan all collections
-clawmem embed                   # Embed all un-embedded fragments
-clawmem embed --force           # Re-embed everything (clears existing vectors)
+clawmem embed                   # Embed all un-embedded fragments (geometry-canary preflight runs first)
+clawmem embed --force           # Re-embed everything (clears existing vectors; aborts BEFORE clearing if the canary preflight fails)
+clawmem embed --force --force-geometry        # v0.21.0: proceed despite a failed/unavailable canary — vault is tainted until a verified rebuild
+clawmem embed --force --recalibrate-canary    # v0.21.0: replace the stored canary baseline after a deliberate model/server change (requires --force)
 ```
 
 ### Conversation Import

package/docs/reference/configuration.md

@@ -38,6 +38,7 @@ Full provider matrix and behavior: [../guides/cloud-embedding.md](../guides/clou
 |---|---|---|
 | `CLAWMEM_PROFILE` | `balanced` | `speed` / `balanced` / `deep`. Sets the kept-score ratio (65% / 55% / 45%), vector timeout, max results, and the `factsTokens` sub-budget. Only `deep` adds query expansion + reranking to the hook path. `speed` makes hooks BM25-only (sub-500ms). |
 | `CLAWMEM_NUDGE_INTERVAL` | `15` | Prompts between lifecycle tool use before a `<vault-nudge>` is injected. `0` to disable. |
+| `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | (superseded) | **No effect since v0.22.0.** The direct-pipeline eval this knob was gated on measured tuned weights at 1/19 hit@1; the direct vector routes now rank by raw cosine instead (see [mcp-tools](mcp-tools.md) → Scoring regimes). Still parsed for backward compatibility — setting it (env or `retrieval.mcp_direct_tuned_weights` in `config.yaml`) logs a once-per-process warning. |
 
 The context-surfacing hook `timeout` is **not** an env var — it lives in `~/.claude/settings.json` (8s default). See [../troubleshooting.md](../troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
 

package/docs/reference/mcp-tools.md

@@ -4,6 +4,12 @@ Complete reference for ClawMem's MCP server tools. These let AI agents search, r
 
 ## Retrieval
 
+**Internal-collection visibility (v0.21.0):** the retrieval tools `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` exclude the system-internal `_clawmem` collection (observations/deductions/handoffs) by default. Opt-ins: pass `includeInternal: true` (all six tools), or — on the tools that expose a `collection` parameter (`search`, `vsearch`, `query`) — name `_clawmem` explicitly in the filter. `find_similar` auto-includes internal results when the REFERENCE document is itself internal. `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are NOT filtered — system memory is their substrate by design.
+
+**Scoring regimes (v0.22.0):** the direct vector routes — `vsearch` and `memory_retrieve`'s semantic/discovery modes — rank non-recency queries by RAW vector cosine. Their `structuredContent` carries `scoreBasis: "vector-cosine"`; raw cosine values are specific to the embedding model that produced them and are NOT comparable across models or to composite scores. Document metadata — including pin — participates only inside groups of exactly-equal raw scores. On these routes `minScore` filters the raw score and has NO default (omitted = no filter; an explicit `0` is honored). Recency-intent queries ("latest…", "recently…", "yesterday…") keep the composite regime with its 0.3 default floor and report `scoreBasis: "composite"`, as do `search`, `query`, `query_plan`, and `memory_retrieve`'s keyword/hybrid/causal/complex modes. `find_similar` has always ranked by raw cosine. Rationale: on the measured vault, raw cosine ranked 16/19 judged targets #1 (MRR 0.912) while the composite stack ranked 1/19 and filtered 14/19 out entirely.
+
+**Degraded vector results (v0.21.0):** under default exclusion the vector scan escalates its depth to fill `limit` with allowed documents, up to a hard cap. When the cap prevents an exhaustive scan and the result is under-filled, `structuredContent` carries `degraded: true` with `degradedReason`: `"excluded-dominant"` (distinct excluded docs account for the shortfall — the guidance line suggests `includeInternal: true` or a refined query) or `"cap-truncation"` (shortfall driven by fragment dedup, neutral guidance). Multi-leg routes (`query`, `query_plan`, `memory_retrieve` complex mode) aggregate `degraded = any(leg)` and list per-leg reasons in `structuredContent.degradedLegs`; single-vector routes (`vsearch`, `find_similar`, `memory_retrieve`'s other modes) report the flat `degraded` + `degradedReason` pair. A small vault whose whole index is scanned without hitting the cap returns a plain short list with NO marker.
+
 ### memory_retrieve
 
 **Recommended entry point.** Auto-classifies query and routes to the optimal backend.
@@ -14,6 +20,7 @@ Complete reference for ClawMem's MCP server tools. These let AI agents search, r
 | `mode` | enum | `auto` | Override: `keyword`, `semantic`, `causal`, `timeline`, `discovery`, `complex`, `hybrid` |
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output (snippets vs full content) |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 Auto-routing:
@@ -35,6 +42,7 @@ Full hybrid pipeline: BM25 + vector + query expansion + cross-encoder reranking.
 | `collection` | string | — | Filter by collection (comma-separated for multi) |
 | `intent` | string | — | Domain hint for ambiguous queries (steers expansion, reranking, chunk selection) |
 | `candidateLimit` | number | 30 | Candidates for reranking (tune precision vs speed) |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 BM25 strong-signal bypass: skips expansion when top BM25 hit >= 0.85 with gap >= 0.15 (disabled when `intent` is provided).
@@ -49,18 +57,21 @@ BM25 only. Zero GPU cost.
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output |
 | `collection` | string | — | Filter by collection |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ### vsearch
 
-Vector only. Semantic similarity.
+Vector only. Semantic similarity. Non-recency queries rank by RAW cosine (`scoreBasis: "vector-cosine"`, v0.22.0); recency-intent queries use the composite regime.
 
 | Param | Type | Default | Description |
 |-------|------|---------|-------------|
 | `query` | string | required | Search query |
 | `limit` | number | 10 | Max results |
+| `minScore` | number | — | Raw-cosine floor on non-recency queries (no default — omitted means no filter; explicit `0` honored). Recency-intent queries keep the composite-scale 0.3 default. |
 | `compact` | boolean | true | Compact output |
 | `collection` | string | — | Filter by collection |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ### intent_search
@@ -84,6 +95,7 @@ Multi-topic decomposition. Use for complex queries spanning multiple subjects.
 | `query` | string | required | Complex or multi-topic query |
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ## Document access
@@ -120,6 +132,7 @@ k-NN vector neighbors of a reference document.
 |-------|------|---------|-------------|
 | `file` | string | required | Path of reference document |
 | `limit` | number | 5 | Max results |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs (auto-included when the reference doc is itself internal) |
 | `vault` | string | — | Named vault |
 
 ### find_causal_links
@@ -168,7 +181,7 @@ Recent session history with handoffs and file changes.
 
 ### memory_pin
 
-Pin a memory for permanent prioritization (+0.3 composite boost).
+Pin a memory: lifecycle retention plus prioritization among relevance-equivalent results. On composite surfaces (hooks, `query`, `search`) pinned docs get the +0.3 composite boost; on the raw vector routes (v0.22.0) pin wins exact raw-score ties but never overrides a relevance difference.
 
 | Param | Type | Default | Description |
 |-------|------|---------|-------------|

package/docs/troubleshooting.md

@@ -93,10 +93,13 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
 - Fix: re-deploy a working reranker — the **zerank-2 seq-cls sidecar** (`extras/rerankers/zerank-2-seq/`, see CLAUDE.md "SOTA upgrade") or the default `qwen3-reranker-0.6B` — and confirm `CLAWMEM_RERANK_URL` points at it. Re-run `clawmem rerank-health`; it should exit 0. For remote-sidecar setups, schedule `clawmem-rerank-health.timer` (CLAUDE.md → Background Services) so a future silent reversion pages you.
 
 **Vector search returns weak or irrelevant results even though embeddings exist**
-- BM25/keyword search works and `doctor` shows vectors present + consistent, but `vsearch`/`find_similar` (and the vector half of `query`) return loosely-related results, or "the same few docs regardless of query." This is an embedding-**quality** problem, not a ClawMem index problem: the model is producing poorly-discriminating vectors. The most common cause is a **server-side pooling/normalization misconfiguration** — e.g. serving a last-token model (Qwen3-Embedding family) without `--pooling last`, or without L2 normalization. Mean-pooling a last-token model gives usable self-retrieval but collapsed semantic separation (paraphrases score ~0.5 instead of ~0.85).
-- Diagnose: embed two paraphrases and one unrelated sentence through your endpoint and compare cosine similarity. A healthy model scores the paraphrase pair > 0.75 and the unrelated pair < 0.45. If paraphrases score ~0.5 with weak separation, the model/serving is the problem, not ClawMem.
-- Fix: launch the embedding `llama-server` with the pooling its model requires (`--pooling last` for Qwen3-Embedding / last-token models) and ensure outputs are L2-normalized. Re-run the diagnostic; once separation is healthy, retrieval recovers. A re-embed is not required if the dimension is unchanged, though `clawmem embed --force` after the fix guarantees a clean rebuild.
-- Aside: auto-generated `_clawmem/` system docs (observations/deductions) sit near the embedding centroid and can dominate pure-vector tools when discrimination is weak — fixing the model is the real remedy, not filtering.
+- BM25/keyword search works and `doctor` shows vectors present + consistent, but `vsearch`/`find_similar` (and the vector half of `query`) return loosely-related results, or "the same few docs regardless of query." This is an embedding-**quality** problem, not a ClawMem index problem: the model is producing poorly-discriminating vectors. Two common causes, both server-side:
+  - **Pooling misconfiguration** — serving a last-token model (Qwen3-Embedding family) without `--pooling last`, or without L2 normalization. Mean-pooling a last-token model gives usable self-retrieval but collapsed semantic separation (paraphrases score ~0.5 instead of ~0.85).
+  - **Missing EOS anchor** — a last-token model whose GGUF conversion lost `tokenizer.ggml.add_eos_token`: the server never appends the terminator the model reads its embedding from, so last-token pooling reads an arbitrary final text token. Signature: similarity tracks how texts END, not what they mean — identical-vocabulary pairs score low; similarity swings with the truncation point; texts sharing a final word score deceptively high; self-similarity stays ~1.0 (so stored-vs-fresh checks pass). Verified real-world: this exact failure served an entire vault ~0.33 on echo pairs while basic-English probes looked healthy.
+- Diagnose: embed two paraphrases and one unrelated sentence through your endpoint and compare cosine similarity. A healthy model scores the paraphrase pair > 0.75 and the unrelated pair < 0.45. Also test a near-identical pair differing only in the final word — it must score HIGH; low means an unanchored last-token readout. Beware the shared-suffix confound when testing terminators by hand: appending ANY common suffix to both texts inflates similarity by last-token identity — only genuine semantic separation (related high AND unrelated low) proves the fix. `clawmem doctor` runs this battery automatically (geometry canary), and `clawmem embed` refuses to build against a failing geometry (override: `--force-geometry`).
+- Fix: launch the embedding `llama-server` with the pooling its model requires (`--pooling last` for Qwen3-Embedding / last-token models), restore the EOS append when the GGUF lost it (`--override-kv tokenizer.ggml.add_eos_token=bool:true` — a no-op when the metadata is already correct), and ensure outputs are L2-normalized. **A full `clawmem embed --force` is REQUIRED after any serving-side pooling/normalization/EOS change**: the old vectors are faithful to the old geometry, and two geometries at the same dimension are mutually incompatible — fresh queries against stale vectors are cosine-meaningless.
+- Note on `_clawmem/` system docs (observations/deductions): since v0.21.0 the MCP retrieval tools exclude the `_clawmem` collection by default (pass `includeInternal: true` to include it; an explicit `collection` filter naming `_clawmem` also overrides). If internal docs dominate results even with a HEALTHY model, that is the composite-scoring floor (system docs carry high confidence and non-decaying recency), not geometry — the default exclusion is the remedy there.
+- Operational: don't run `embed --force` while the file watcher is indexing bulk changes — stop the watcher or expect write contention (the run now survives transient `SQLITE_BUSY`, but contention still slows it). If you pipe embed output through `tee`, remember the pipeline exit code is `tee`'s — check `PIPESTATUS[0]` for the embed's own status.
 
 **kg_query returns empty for every entity**
 - `entity_triples` is populated by the decision-extractor Stop hook from observer-emitted `<triples>` blocks. Zero rows typically means either (a) the Stop hook has never fired in this vault, or (b) the observer LLM is not emitting `<triples>` blocks.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clawmem",
-  "version": "0.20.2",
+  "version": "0.22.0",
   "description": "On-device memory layer for AI agents. Claude Code, OpenClaw, and Hermes. Hooks + MCP server + hybrid RAG search.",
   "type": "module",
   "bin": {

package/src/busy-retry.ts

@@ -0,0 +1,34 @@
+/**
+ * SQLITE_BUSY detection + bounded ASYNC retry (VSEARCH-TRUST-HARDENING (f).2).
+ *
+ * The awaits between attempts let the event loop breathe so the embed-lease heartbeat
+ * (a setInterval on the same loop) fires; the caller's leaseLost flag is re-checked
+ * between attempts so a reclaimed lease ABORTS instead of writing as a zombie holder.
+ * Retry scope is SQLITE_BUSY only — every other error (including FatalVectorError
+ * dimension/model/schema aborts) propagates unchanged on the first throw.
+ */
+import { EmbedLeaseLostError } from "./store.ts";
+
+const SQLITE_BUSY_RE = /database is locked|SQLITE_BUSY/i;
+
+export function isSqliteBusyError(err: unknown): boolean {
+  return err instanceof Error && SQLITE_BUSY_RE.test(err.message);
+}
+
+export async function retryOnBusyAsync<T>(
+  fn: () => T,
+  label: string,
+  isLeaseLost: () => boolean,
+  opts?: { delaysMs?: number[]; onRetry?: (label: string, attempt: number, delayMs: number) => void }
+): Promise<T> {
+  const delays = opts?.delaysMs ?? [1000, 2000, 4000];
+  for (let attempt = 0; ; attempt++) {
+    try { return fn(); }
+    catch (err) {
+      if (!isSqliteBusyError(err) || attempt >= delays.length) throw err;
+      opts?.onRetry?.(label, attempt + 1, delays[attempt]!);
+      await new Promise(res => setTimeout(res, delays[attempt]!));
+      if (isLeaseLost()) throw new EmbedLeaseLostError();
+    }
+  }
+}