clawmem 0.20.2 → 0.21.0

package/AGENTS.md

@@ -59,7 +59,7 @@ clawmem doctor                              # full health check (clawmem status
 | `feedback-loop` | Stop | tracks referenced notes → confidence boosts, co-activations, utility signals |
 
 **Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
-**Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
+**Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3. Note the MCP retrieval tools themselves exclude `_clawmem` by default since v0.21.0 — pass `includeInternal: true` when system-internal memory (observations/handoffs/deductions) is the target.
 **Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) + an activation floor; only `deep` adds query expansion + reranking to the hook path. → [docs/concepts/hooks-vs-mcp.md](docs/concepts/hooks-vs-mcp.md).
 
 ### Tier 3 — agent-initiated (one targeted MCP call)

package/README.md

@@ -531,6 +531,8 @@ clawmem status                                  Quick index status
 
 Registered by `clawmem setup mcp`. Available to any MCP-compatible client.
 
+**Internal-collection visibility (v0.21.0):** the retrieval tools `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` exclude the system-internal `_clawmem` collection by default — pass `includeInternal: true` (or name `_clawmem` in an explicit `collection` filter where the tool has one) to include it. `find_similar` auto-includes internal neighbors when the reference document is itself internal; `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are unfiltered by design. Full contract (including the `degraded` under-fill markers): [docs/reference/mcp-tools.md](docs/reference/mcp-tools.md).
+
 | Tool | Description |
 |---|---|
 | `__IMPORTANT` | Workflow guide: prefer `memory_retrieve` → match query type to tool → `multi_get` for full content |
@@ -789,6 +791,7 @@ Notes referenced by the agent during a session get boosted (`access_count++`). U
 | `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server URL. Without it, falls to `node-llama-cpp` (if allowed). |
 | `CLAWMEM_RERANK_API_KEY` | (none) | Bearer token for an authenticated remote reranker endpoint. Independent of the embed and LLM keys. |
 | `CLAWMEM_NO_LOCAL_MODELS` | `false` | Block `node-llama-cpp` from auto-downloading GGUF models. Set `true` for remote-only setups where you want fail-fast on unreachable endpoints. |
+| `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | `false` | **v0.21.0.** When `true`, the MCP direct tools (`search`, `vsearch`, `memory_retrieve`) score non-recency queries with the retrieval-tuned `query`-tool weights. Ships off — the eval evidence covered only the hybrid `query` pipeline. YAML equivalent: `retrieval.mcp_direct_tuned_weights` (env wins). |
 | `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 consolidation merge-safety threshold when candidate and existing anchors align. Merges above this normalized 3-gram cosine score are allowed. |
 | `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety threshold — fallback when anchor sets are ambiguous. |
 | `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration before enabling the gate. |

package/SKILL.md

@@ -35,7 +35,7 @@ Hooks handle ~90% of retrieval at zero agent effort.
 
 **Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
 
-**Hook blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, and cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
+**Hook blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, and cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3. Note the MCP retrieval tools themselves exclude `_clawmem` by default since v0.21.0 — pass `includeInternal: true` when system-internal memory (observations/handoffs/deductions) is the target.
 
 **Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) and an activation floor. Only `deep` adds query expansion + reranking to the hook path. Profile and the hook `timeout` are set in `~/.claude/settings.json` — see *Operational gotchas* for timeout tuning.
 

package/docs/concepts/composite-scoring.md

@@ -19,7 +19,9 @@ compositeScore = (w_search * searchScore + w_recency * recencyScore + w_confiden
 
 Recency intent is detected automatically when queries contain "latest", "recent", "last session", etc. — and **takes precedence over the `query`-tool weights**: a recency-phrased `query` call still uses the Recency-intent column.
 
-The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. All other tools (`search`, `vsearch`, `memory_retrieve`, and the context-surfacing hook) use the **Normal** column.
+The **`query` tool** uses retrieval-tuned weights (search 0.70) derived from a held-out judged-relevance eval (v0.13.0) — more weight on topical relevance measurably improves graded NDCG@10 without demoting the newest-correct version of evolving docs. `search`, `vsearch`, `memory_retrieve`, and the context-surfacing hook **keep the Normal column by default**. Since v0.21.0 a config knob — `retrieval.mcp_direct_tuned_weights: true` (or env `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS=true`) — switches the three MCP direct tools' non-recency scoring to the `query`-tool column; it ships **off** because the eval evidence covered only the hybrid `query` pipeline, and the flip is gated on a direct-pipeline eval (see [configuration](../reference/configuration.md)).
+
+Threshold note: EOS-anchored last-token embedding models (e.g. zembed-1 served correctly) produce a **compressed-high similarity band** — unrelated pairs sit near ~0.4 rather than ~0.2. Relative ordering is what matters; treat absolute `minScore` cutoffs as model-dependent rather than universal.
 
 ## Signal breakdown
 

package/docs/guides/inference-services.md

@@ -60,9 +60,18 @@ ZeroEntropy's distillation-paired stack — best retrieval quality, total ~16 GB
 | Reranker | 8090 | [zerank-2 seq-cls sidecar](../../extras/rerankers/zerank-2-seq/) (transformers, bf16) | ~9 GB | SOTA reranker — **not** a GGUF |
 
 ```bash
-# Embedding (zembed-1) — -ub MUST equal -b for non-causal attention
+# Embedding (zembed-1) — -ub MUST equal -b for non-causal attention.
+# --pooling last: zembed-1 is a last-token model; declare it explicitly.
+# --override-kv ...add_eos_token: last-token models read the embedding at their EOS
+# anchor. If the GGUF conversion lost add_eos_token, the server never appends the
+# terminator and similarity collapses to last-token identity (see troubleshooting →
+# "Vector search returns weak or irrelevant results"). The override restores it and
+# is a no-op when the metadata is already correct. Do NOT add these two flags to
+# mean-pooling models like the default EmbeddingGemma.
 llama-server -m zembed-1-Q4_K_M.gguf \
-  --embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
+  --embeddings --pooling last \
+  --override-kv tokenizer.ggml.add_eos_token=bool:true \
+  --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
 
 # Reranker (zerank-2) — seq-cls SIDECAR (transformers, bf16), NOT a llama-server GGUF:
 cd extras/rerankers/zerank-2-seq

package/docs/guides/upgrading.md

@@ -1,6 +1,6 @@
 # Upgrading ClawMem
 
-Guide for upgrading between released versions. Current: **v0.10.4**.
+Guide for upgrading between released versions. Current: **v0.21.0**.
 
 ClawMem upgrades are designed to be drop-in: pull the new version, restart any long-lived processes, and the SQLite schema auto-migrates on first open. This guide documents per-version specifics for upgrades that have additional considerations beyond the quick path below.
 
@@ -59,6 +59,24 @@ docker compose up -d reranker                      # /v1/rerank on :8090
 
 ---
 
+## v0.21.0: MCP internal-collection exclusion + embedding-geometry canary
+
+No migration command required. Two things to know:
+
+**Behavior change — `_clawmem` excluded from MCP retrieval by default.** `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` no longer return the system-internal `_clawmem` collection (observations/deductions/handoffs) unless asked. If a workflow depended on those appearing in MCP results, pass `includeInternal: true` or name `_clawmem` in an explicit `collection` filter. `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are unfiltered by design, and hooks already filtered internal docs — `<vault-context>` behavior is unchanged. Details: [docs/reference/mcp-tools.md](../reference/mcp-tools.md).
+
+**What auto-applies on first open** (additive, idempotent — same contract as prior releases): the `embed_canary` and `vault_flags` tables, and an `embed_input_fp` column on `content_vectors`.
+
+**No re-embed required.** Existing vectors stay valid. Pre-0.21.0 rows carry no input fingerprint, so `clawmem doctor`'s new sampled vector validation checks them structurally and flags title provenance as unavailable ("legacy") until each document's next natural re-embed — informational, not an error.
+
+**The canary baseline seeds itself.** The first `clawmem embed` run after upgrade (including a timer-fired one) probes the embedding server with a pair-separation battery and persists a first-healthy baseline; subsequent runs alert relative to it, and a broken-geometry server now aborts `embed --force` BEFORE anything is cleared. `--force-geometry` (proceed despite a failed probe; the vault is flagged tainted until a verified rebuild) and `--force --recalibrate-canary` (replace the baseline after a deliberate model/server change) are operator overrides, not upgrade steps.
+
+**Opt-in knob:** `retrieval.mcp_direct_tuned_weights` (config) / `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` (env), default `false` — scores the MCP direct tools' non-recency queries with the retrieval-tuned `query`-tool weights. Leave it off unless you have direct-pipeline eval evidence for your vault.
+
+Upgrades from v0.13 → v0.20 shipped no steps beyond the [quick path](#quick-path) — schema changes auto-apply; see [RELEASE_NOTES.md](../../RELEASE_NOTES.md) for what each version changed.
+
+---
+
 ## v0.12.0: query reranking blend (no action required)
 
 v0.12.0 changes the `query` tool's rerank/RRF blend so the cross-encoder reranker can promote the best document to the top — the previous blend left RRF #1 mathematically immovable. It is a pure ranking-quality change: **no migration, no schema change, no config change.** It applies automatically on upgrade. Hooks and the per-session MCP stdio server pick it up on their next invocation; restart any long-lived `query` host — `clawmem serve` and persistent MCP/daemon processes — to pick up the improved ordering. The default reranker (`qwen3-reranker-0.6B`) is unchanged, and the blend improvement applies whatever reranker `:8090` serves. (`intent_search` and the context-surfacing hook keep their existing blends.)
@@ -145,7 +163,7 @@ Hooks and the stdio MCP server pick up the new binary automatically on their nex
 - **`plugins.slots.memory: "clawmem"` replaces `plugins.slots.contextEngine: "clawmem"`.** On the new pure-memory plugin, the exclusive slot is `memory`. The `setup openclaw` next-steps output tells you to run `openclaw plugins enable clawmem`, which sets the slot and disables competing memory plugins (`memory-core`, `memory-lancedb`) in a single command. You do NOT need to run the older `openclaw config set plugins.slots.contextEngine clawmem` pattern on v0.10.0.
 - **`src/openclaw/package.json` is now the plugin's discovery manifest.** OpenClaw v2026.4.11's `discoverInDirectory` reads `package.json` for the `openclaw.extensions` field and uses that to decide whether a directory under `~/.openclaw/extensions/` is a valid plugin. The older `openclaw.plugin.json` manifest is still shipped and parsed at runtime, but it is not sufficient to pass discovery on v2026.4.11+ without the `package.json` companion file. v0.10.0 adds the `package.json` to the plugin source tree, and `clawmem setup openclaw` verifies it is present before copying.
 - **`clawmem setup openclaw` defaults to recursive copy instead of symlink.** OpenClaw v2026.4.11 walks `~/.openclaw/extensions/` with `readdirSync({ withFileTypes: true })` and uses `dirent.isDirectory()` to descend into candidate plugin directories. Symlinks to directories report `isDirectory() === false` on that API shape, so a symlinked plugin is silently skipped during discovery. v0.10.0's `cmdSetupOpenClaw` therefore copies the plugin source into `~/.openclaw/extensions/clawmem/` with `cpSync(..., { recursive: true, dereference: true })`. A `--link` opt-in flag preserves the old symlink behavior for local development and for older OpenClaw versions, with a warning that v2026.4.11+ discovery will skip the symlink. Setup is idempotent: any existing plugin directory or stale symlink is removed before the new copy is written.
-- **Multi-user ownership check (OpenClaw v2026.4.11+).** If the gateway runs as a dedicated system user (e.g. `openclaw`) and you run `clawmem setup openclaw` as a different user (e.g. `sciros`), the copied plugin directory is owned by the installer user, but OpenClaw's ownership check rejects it with `suspicious ownership (uid=1001, expected uid=997 or root)`. This is a security feature that prevents a privileged gateway process from loading code a less-privileged user dropped into its extensions directory. Fix: `sudo chown -R <gateway-user>:<gateway-group> ~/.openclaw/extensions/clawmem`. Single-user installs where you ARE the gateway user are not affected — your own user owns the copy, and the ownership check passes.
+- **Multi-user ownership check (OpenClaw v2026.4.11+).** If the gateway runs as a dedicated system user (e.g. `openclaw`) and you run `clawmem setup openclaw` as a different user (e.g. `alice`), the copied plugin directory is owned by the installer user, but OpenClaw's ownership check rejects it with `suspicious ownership (uid=1001, expected uid=997 or root)`. This is a security feature that prevents a privileged gateway process from loading code a less-privileged user dropped into its extensions directory. Fix: `sudo chown -R <gateway-user>:<gateway-group> ~/.openclaw/extensions/clawmem`. Single-user installs where you ARE the gateway user are not affected — your own user owns the copy, and the ownership check passes.
 
 ### Rollback
 
@@ -191,7 +209,7 @@ Hooks + MCP stdio pick up new code automatically on next invocation — no resta
 
 When the user's prompt mentions entities already known to the vault (via `entity_nodes`), `context-surfacing` now appends a token-bounded `<vault-facts>` block of raw SPO triple lines to `<vault-context>`, alongside the existing `<facts>` / `<relationships>` blocks. This feeds the model current-state knowledge about entities the user is talking about, without requiring the agent to call `kg_query` explicitly.
 
-- **Three-path entity seeding** — canonical-ID regex (e.g. `default:project:clawmem`) → proper-noun extraction via `resolveEntityTypeExact` → longer-first n-gram scan (3-gram > 2-gram > 1-gram) for lowercase/hyphenated vocabulary like `forge-stack`, `oauth2`, `vm 202`. All three paths run prompt-only — entity seeds NEVER come from surfaced doc bodies, so topic-boosted off-topic docs cannot pollute the facts block.
+- **Three-path entity seeding** — canonical-ID regex (e.g. `default:project:clawmem`) → proper-noun extraction via `resolveEntityTypeExact` → longer-first n-gram scan (3-gram > 2-gram > 1-gram) for lowercase/hyphenated vocabulary like `side-project`, `oauth2`, `vm 12`. All three paths run prompt-only — entity seeds NEVER come from surfaced doc bodies, so topic-boosted off-topic docs cannot pollute the facts block.
 - **Profile-gated token sub-budget** — `factsTokens=0` on `speed` disables the stage entirely. `balanced` uses 200 tokens, `deep` uses 250. The sub-budget is dedicated — `<vault-facts>` cannot steal budget from `<facts>` or `<relationships>`.
 - **Truncation** — at the triple boundary, never mid-triple, never emits an empty block.
 - **Fail-open** — empty entity set → skip. Budget too small → drop block. Per-entity DB error → skip that entity. Any exception in the stage → return baseline `vault-context` unchanged.

package/docs/quickstart.md

@@ -70,7 +70,7 @@ llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
   --reranking --port 8090 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 512
 ```
 
-> **SOTA upgrade (16GB+ GPU):** Replace embedding with zembed-1-Q4_K_M (2560d, `-b 2048 -ub 2048`). For the reranker, use the **zerank-2 seq-cls sidecar** (transformers, bf16) at [`extras/rerankers/zerank-2-seq/`](../extras/rerankers/zerank-2-seq/) — the `zerank-2-Q4_K_M` GGUF is deprecated (llama.cpp drops its score head). See [the inference services guide](guides/inference-services.md). **CC-BY-NC-4.0** — non-commercial only.
+> **SOTA upgrade (16GB+ GPU):** Replace embedding with zembed-1-Q4_K_M (2560d, `-b 2048 -ub 2048`, **plus `--pooling last --override-kv tokenizer.ggml.add_eos_token=bool:true`** — last-token models need their EOS anchor; the exact launch line is in [the inference services guide](guides/inference-services.md)). For the reranker, use the **zerank-2 seq-cls sidecar** (transformers, bf16) at [`extras/rerankers/zerank-2-seq/`](../extras/rerankers/zerank-2-seq/) — the `zerank-2-Q4_K_M` GGUF is deprecated (llama.cpp drops its score head). **CC-BY-NC-4.0** — non-commercial only.
 
 See [GPU services guide](guides/systemd-services.md) for systemd setup and remote GPU configuration.
 

package/docs/reference/cli.md

@@ -7,7 +7,7 @@ Complete command reference for the ClawMem memory engine. Always use the `bin/cl
 ```bash
 clawmem init                    # Initialize vault (creates SQLite DB)
 clawmem status                  # Quick index status
-clawmem doctor                  # Full health check (GPU connectivity, index integrity)
+clawmem doctor                  # Full health check (GPU connectivity, index integrity, embedding-geometry canary, sampled vector validation)
 ```
 
 ## Collection management
@@ -30,8 +30,10 @@ clawmem mine <dir> --dry-run                   # Preview without importing
 clawmem mine <dir> --synthesize                # v0.7.2: import + post-import LLM fact extraction
 clawmem mine <dir> --synthesize --synthesis-max-docs 50   # Cap synthesis to first 50 conversations (default 20)
 clawmem reindex                                # Force re-scan all collections
-clawmem embed                   # Embed all un-embedded fragments
-clawmem embed --force           # Re-embed everything (clears existing vectors)
+clawmem embed                   # Embed all un-embedded fragments (geometry-canary preflight runs first)
+clawmem embed --force           # Re-embed everything (clears existing vectors; aborts BEFORE clearing if the canary preflight fails)
+clawmem embed --force --force-geometry        # v0.21.0: proceed despite a failed/unavailable canary — vault is tainted until a verified rebuild
+clawmem embed --force --recalibrate-canary    # v0.21.0: replace the stored canary baseline after a deliberate model/server change (requires --force)
 ```
 
 ### Conversation Import

package/docs/reference/configuration.md

@@ -38,6 +38,7 @@ Full provider matrix and behavior: [../guides/cloud-embedding.md](../guides/clou
 |---|---|---|
 | `CLAWMEM_PROFILE` | `balanced` | `speed` / `balanced` / `deep`. Sets the kept-score ratio (65% / 55% / 45%), vector timeout, max results, and the `factsTokens` sub-budget. Only `deep` adds query expansion + reranking to the hook path. `speed` makes hooks BM25-only (sub-500ms). |
 | `CLAWMEM_NUDGE_INTERVAL` | `15` | Prompts between lifecycle tool use before a `<vault-nudge>` is injected. `0` to disable. |
+| `CLAWMEM_MCP_DIRECT_TUNED_WEIGHTS` | `false` | When `true`, the MCP direct tools (`search`, `vsearch`, `memory_retrieve`) score non-recency queries with the retrieval-tuned `query`-tool weights (search 0.70) instead of the Normal weights. Ships **off**: the eval evidence covered only the hybrid `query` pipeline; flipping the default is gated on a direct-pipeline eval. YAML equivalent in `config.yaml`: `retrieval.mcp_direct_tuned_weights: true` (env wins). |
 
 The context-surfacing hook `timeout` is **not** an env var — it lives in `~/.claude/settings.json` (8s default). See [../troubleshooting.md](../troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
 

package/docs/reference/mcp-tools.md

@@ -4,6 +4,10 @@ Complete reference for ClawMem's MCP server tools. These let AI agents search, r
 
 ## Retrieval
 
+**Internal-collection visibility (v0.21.0):** the retrieval tools `search`, `vsearch`, `query`, `query_plan`, `memory_retrieve`, and `find_similar` exclude the system-internal `_clawmem` collection (observations/deductions/handoffs) by default. Opt-ins: pass `includeInternal: true` (all six tools), or — on the tools that expose a `collection` parameter (`search`, `vsearch`, `query`) — name `_clawmem` explicitly in the filter. `find_similar` auto-includes internal results when the REFERENCE document is itself internal. `intent_search`, `find_causal_links`, `kg_query`, `session_log`, and `timeline` are NOT filtered — system memory is their substrate by design.
+
+**Degraded vector results (v0.21.0):** under default exclusion the vector scan escalates its depth to fill `limit` with allowed documents, up to a hard cap. When the cap prevents an exhaustive scan and the result is under-filled, `structuredContent` carries `degraded: true` with `degradedReason`: `"excluded-dominant"` (distinct excluded docs account for the shortfall — the guidance line suggests `includeInternal: true` or a refined query) or `"cap-truncation"` (shortfall driven by fragment dedup, neutral guidance). Multi-leg routes (`query`, `query_plan`, `memory_retrieve` complex mode) aggregate `degraded = any(leg)` and list per-leg reasons in `structuredContent.degradedLegs`; single-vector routes (`vsearch`, `find_similar`, `memory_retrieve`'s other modes) report the flat `degraded` + `degradedReason` pair. A small vault whose whole index is scanned without hitting the cap returns a plain short list with NO marker.
+
 ### memory_retrieve
 
 **Recommended entry point.** Auto-classifies query and routes to the optimal backend.
@@ -14,6 +18,7 @@ Complete reference for ClawMem's MCP server tools. These let AI agents search, r
 | `mode` | enum | `auto` | Override: `keyword`, `semantic`, `causal`, `timeline`, `discovery`, `complex`, `hybrid` |
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output (snippets vs full content) |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 Auto-routing:
@@ -35,6 +40,7 @@ Full hybrid pipeline: BM25 + vector + query expansion + cross-encoder reranking.
 | `collection` | string | — | Filter by collection (comma-separated for multi) |
 | `intent` | string | — | Domain hint for ambiguous queries (steers expansion, reranking, chunk selection) |
 | `candidateLimit` | number | 30 | Candidates for reranking (tune precision vs speed) |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 BM25 strong-signal bypass: skips expansion when top BM25 hit >= 0.85 with gap >= 0.15 (disabled when `intent` is provided).
@@ -49,6 +55,7 @@ BM25 only. Zero GPU cost.
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output |
 | `collection` | string | — | Filter by collection |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ### vsearch
@@ -61,6 +68,7 @@ Vector only. Semantic similarity.
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output |
 | `collection` | string | — | Filter by collection |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ### intent_search
@@ -84,6 +92,7 @@ Multi-topic decomposition. Use for complex queries spanning multiple subjects.
 | `query` | string | required | Complex or multi-topic query |
 | `limit` | number | 10 | Max results |
 | `compact` | boolean | true | Compact output |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs |
 | `vault` | string | — | Named vault |
 
 ## Document access
@@ -120,6 +129,7 @@ k-NN vector neighbors of a reference document.
 |-------|------|---------|-------------|
 | `file` | string | required | Path of reference document |
 | `limit` | number | 5 | Max results |
+| `includeInternal` | boolean | false | Include system-internal `_clawmem` docs (auto-included when the reference doc is itself internal) |
 | `vault` | string | — | Named vault |
 
 ### find_causal_links

package/docs/troubleshooting.md

@@ -93,10 +93,13 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
 - Fix: re-deploy a working reranker — the **zerank-2 seq-cls sidecar** (`extras/rerankers/zerank-2-seq/`, see CLAUDE.md "SOTA upgrade") or the default `qwen3-reranker-0.6B` — and confirm `CLAWMEM_RERANK_URL` points at it. Re-run `clawmem rerank-health`; it should exit 0. For remote-sidecar setups, schedule `clawmem-rerank-health.timer` (CLAUDE.md → Background Services) so a future silent reversion pages you.
 
 **Vector search returns weak or irrelevant results even though embeddings exist**
-- BM25/keyword search works and `doctor` shows vectors present + consistent, but `vsearch`/`find_similar` (and the vector half of `query`) return loosely-related results, or "the same few docs regardless of query." This is an embedding-**quality** problem, not a ClawMem index problem: the model is producing poorly-discriminating vectors. The most common cause is a **server-side pooling/normalization misconfiguration** — e.g. serving a last-token model (Qwen3-Embedding family) without `--pooling last`, or without L2 normalization. Mean-pooling a last-token model gives usable self-retrieval but collapsed semantic separation (paraphrases score ~0.5 instead of ~0.85).
-- Diagnose: embed two paraphrases and one unrelated sentence through your endpoint and compare cosine similarity. A healthy model scores the paraphrase pair > 0.75 and the unrelated pair < 0.45. If paraphrases score ~0.5 with weak separation, the model/serving is the problem, not ClawMem.
-- Fix: launch the embedding `llama-server` with the pooling its model requires (`--pooling last` for Qwen3-Embedding / last-token models) and ensure outputs are L2-normalized. Re-run the diagnostic; once separation is healthy, retrieval recovers. A re-embed is not required if the dimension is unchanged, though `clawmem embed --force` after the fix guarantees a clean rebuild.
-- Aside: auto-generated `_clawmem/` system docs (observations/deductions) sit near the embedding centroid and can dominate pure-vector tools when discrimination is weak — fixing the model is the real remedy, not filtering.
+- BM25/keyword search works and `doctor` shows vectors present + consistent, but `vsearch`/`find_similar` (and the vector half of `query`) return loosely-related results, or "the same few docs regardless of query." This is an embedding-**quality** problem, not a ClawMem index problem: the model is producing poorly-discriminating vectors. Two common causes, both server-side:
+  - **Pooling misconfiguration** — serving a last-token model (Qwen3-Embedding family) without `--pooling last`, or without L2 normalization. Mean-pooling a last-token model gives usable self-retrieval but collapsed semantic separation (paraphrases score ~0.5 instead of ~0.85).
+  - **Missing EOS anchor** — a last-token model whose GGUF conversion lost `tokenizer.ggml.add_eos_token`: the server never appends the terminator the model reads its embedding from, so last-token pooling reads an arbitrary final text token. Signature: similarity tracks how texts END, not what they mean — identical-vocabulary pairs score low; similarity swings with the truncation point; texts sharing a final word score deceptively high; self-similarity stays ~1.0 (so stored-vs-fresh checks pass). Verified real-world: this exact failure served an entire vault ~0.33 on echo pairs while basic-English probes looked healthy.
+- Diagnose: embed two paraphrases and one unrelated sentence through your endpoint and compare cosine similarity. A healthy model scores the paraphrase pair > 0.75 and the unrelated pair < 0.45. Also test a near-identical pair differing only in the final word — it must score HIGH; low means an unanchored last-token readout. Beware the shared-suffix confound when testing terminators by hand: appending ANY common suffix to both texts inflates similarity by last-token identity — only genuine semantic separation (related high AND unrelated low) proves the fix. `clawmem doctor` runs this battery automatically (geometry canary), and `clawmem embed` refuses to build against a failing geometry (override: `--force-geometry`).
+- Fix: launch the embedding `llama-server` with the pooling its model requires (`--pooling last` for Qwen3-Embedding / last-token models), restore the EOS append when the GGUF lost it (`--override-kv tokenizer.ggml.add_eos_token=bool:true` — a no-op when the metadata is already correct), and ensure outputs are L2-normalized. **A full `clawmem embed --force` is REQUIRED after any serving-side pooling/normalization/EOS change**: the old vectors are faithful to the old geometry, and two geometries at the same dimension are mutually incompatible — fresh queries against stale vectors are cosine-meaningless.
+- Note on `_clawmem/` system docs (observations/deductions): since v0.21.0 the MCP retrieval tools exclude the `_clawmem` collection by default (pass `includeInternal: true` to include it; an explicit `collection` filter naming `_clawmem` also overrides). If internal docs dominate results even with a HEALTHY model, that is the composite-scoring floor (system docs carry high confidence and non-decaying recency), not geometry — the default exclusion is the remedy there.
+- Operational: don't run `embed --force` while the file watcher is indexing bulk changes — stop the watcher or expect write contention (the run now survives transient `SQLITE_BUSY`, but contention still slows it). If you pipe embed output through `tee`, remember the pipeline exit code is `tee`'s — check `PIPESTATUS[0]` for the embed's own status.
 
 **kg_query returns empty for every entity**
 - `entity_triples` is populated by the decision-extractor Stop hook from observer-emitted `<triples>` blocks. Zero rows typically means either (a) the Stop hook has never fired in this vault, or (b) the observer LLM is not emitting `<triples>` blocks.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clawmem",
-  "version": "0.20.2",
+  "version": "0.21.0",
   "description": "On-device memory layer for AI agents. Claude Code, OpenClaw, and Hermes. Hooks + MCP server + hybrid RAG search.",
   "type": "module",
   "bin": {

package/src/busy-retry.ts

@@ -0,0 +1,34 @@
+/**
+ * SQLITE_BUSY detection + bounded ASYNC retry (VSEARCH-TRUST-HARDENING (f).2).
+ *
+ * The awaits between attempts let the event loop breathe so the embed-lease heartbeat
+ * (a setInterval on the same loop) fires; the caller's leaseLost flag is re-checked
+ * between attempts so a reclaimed lease ABORTS instead of writing as a zombie holder.
+ * Retry scope is SQLITE_BUSY only — every other error (including FatalVectorError
+ * dimension/model/schema aborts) propagates unchanged on the first throw.
+ */
+import { EmbedLeaseLostError } from "./store.ts";
+
+const SQLITE_BUSY_RE = /database is locked|SQLITE_BUSY/i;
+
+export function isSqliteBusyError(err: unknown): boolean {
+  return err instanceof Error && SQLITE_BUSY_RE.test(err.message);
+}
+
+export async function retryOnBusyAsync<T>(
+  fn: () => T,
+  label: string,
+  isLeaseLost: () => boolean,
+  opts?: { delaysMs?: number[]; onRetry?: (label: string, attempt: number, delayMs: number) => void }
+): Promise<T> {
+  const delays = opts?.delaysMs ?? [1000, 2000, 4000];
+  for (let attempt = 0; ; attempt++) {
+    try { return fn(); }
+    catch (err) {
+      if (!isSqliteBusyError(err) || attempt >= delays.length) throw err;
+      opts?.onRetry?.(label, attempt + 1, delays[attempt]!);
+      await new Promise(res => setTimeout(res, delays[attempt]!));
+      if (isLeaseLost()) throw new EmbedLeaseLostError();
+    }
+  }
+}