clawmem 0.14.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -1,789 +1,238 @@
1
1
  # ClawMem — Agent Quick Reference
2
2
 
3
- ## Inference Services
3
+ On-device retrieval-augmented memory for Claude Code, OpenClaw, and Hermes agents. **Hooks** auto-inject context (~90%); **MCP tools** cover targeted recall (~10%). TypeScript on Bun, MIT.
4
4
 
5
- ClawMem uses three inference services (embedding, LLM, reranker). The **default** stack runs all three as `llama-server` instances; the **SOTA** stack swaps the reranker for a transformers sidecar (same `/v1/rerank` contract). The `bin/clawmem` wrapper points at `localhost:8088/8089/8090`.
5
+ This file is the **lean root SSOT** agent-facing essentials only. Deep reference lives in `docs/` (linked throughout, indexed at the bottom). `CLAUDE.md` is an `@AGENTS.md` import; `SKILL.md` is the portable on-demand operating reference.
6
6
 
7
- **Default (QMD native combo, any GPU or in-process):**
8
-
9
- | Service | Port | Model | VRAM | Protocol |
10
- |---|---|---|---|---|
11
- | Embedding | 8088 | EmbeddingGemma-300M-Q8_0 | ~400MB | `/v1/embeddings` |
12
- | LLM | 8089 | qmd-query-expansion-1.7B-q4_k_m | ~2.2GB | `/v1/chat/completions` |
13
- | Reranker | 8090 | qwen3-reranker-0.6B-Q8_0 | ~1.3GB | `/v1/rerank` |
14
-
15
- All three models auto-download via `node-llama-cpp` if no server is running (Metal on Apple Silicon, Vulkan where available, CPU as last resort). Fast with GPU acceleration (Metal/Vulkan); significantly slower on CPU-only.
16
-
17
- **SOTA upgrade (16GB+ GPU):** CC-BY-NC-4.0 — non-commercial only.
18
-
19
- | Service | Port | Model | VRAM | Protocol |
20
- |---|---|---|---|---|
21
- | Embedding | 8088 | zembed-1-Q4_K_M | ~4.4GB | `/v1/embeddings` |
22
- | LLM | 8089 | qmd-query-expansion-1.7B-q4_k_m | ~2.2GB | `/v1/chat/completions` |
23
- | Reranker | 8090 | zerank-2 seq-cls sidecar | ~9GB (bf16) | `/v1/rerank` |
24
-
25
- zembed-1 (2560d, 32K context, SOTA retrieval) distilled from zerank-2 via zELO — optimal pairing. The SOTA reranker is the **zerank-2 seq-cls sidecar** at `extras/rerankers/zerank-2-seq/` (transformers, bf16, ~9GB VRAM) — **not** a GGUF. The older `zerank-2-Q4_K_M` GGUF is **deprecated**: llama.cpp drops zerank's score head, so its scores are near-zero and uninformative (final ordering stays RRF-dominated). Full SOTA stack ≈ 16GB VRAM.
26
-
27
- **Remote option:** Set `CLAWMEM_EMBED_URL`, `CLAWMEM_LLM_URL`, `CLAWMEM_RERANK_URL` to the remote host. Set `CLAWMEM_NO_LOCAL_MODELS=true` to prevent surprise fallback downloads.
28
-
29
- **Cloud embedding:** Set `CLAWMEM_EMBED_API_KEY` + `CLAWMEM_EMBED_URL` + `CLAWMEM_EMBED_MODEL` to use a cloud provider instead of local GPU. Supported: Jina AI (recommended: `jina-embeddings-v5-text-small`, 1024d), OpenAI, Voyage, Cohere. Cloud mode enables batch embedding (50 frags/request), provider-specific retrieval params auto-detected from URL (Jina `task`, Voyage/Cohere `input_type`), server-side truncation, and adaptive TPM-aware pacing. Set `CLAWMEM_EMBED_TPM_LIMIT` to match your tier.
30
-
31
- **Qwen3 /no_think flag:** Qwen3 uses thinking tokens by default. ClawMem appends `/no_think` to all prompts automatically for structured output.
32
-
33
- ### Model Recommendations
34
-
35
- | Role | Default (QMD native) | SOTA Upgrade | Notes |
36
- |---|---|---|---|
37
- | Embedding | [EmbeddingGemma-300M-Q8_0](https://huggingface.co/ggml-org/embeddinggemma-300M-GGUF) (314MB, 768d) | [zembed-1-Q4_K_M](https://huggingface.co/Abhiray/zembed-1-Q4_K_M-GGUF) (2.4GB, 2560d) | zembed-1: 32K context, SOTA retrieval. `-ub` must match `-b`. |
38
- | LLM | [qmd-query-expansion-1.7B-q4_k_m](https://huggingface.co/tobil/qmd-query-expansion-1.7B-gguf) (~1.1GB) | Same | QMD's Qwen3-1.7B finetune for query expansion. |
39
- | Reranker | [qwen3-reranker-0.6B-Q8_0](https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF) (~600MB) | [zerank-2 seq-cls sidecar](extras/rerankers/zerank-2-seq/) (bf16, ~9GB) | zerank-2: NDCG@10 ahead of Cohere rerank-3.5. Served via transformers sidecar, **not** GGUF (the GGUF drops the score head → inert). CC-BY-NC. |
40
-
41
- ### Server Setup (all three use llama-server)
42
-
43
- ```bash
44
- # === Default (QMD native combo) ===
45
-
46
- # Embedding (--embeddings flag required)
47
- llama-server -m embeddinggemma-300M-Q8_0.gguf \
48
- --embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 2048
49
-
50
- # LLM (QMD finetuned model)
51
- llama-server -m qmd-query-expansion-1.7B-q4_k_m.gguf \
52
- --port 8089 --host 0.0.0.0 -ngl 99 -c 4096 --batch-size 512
53
-
54
- # Reranker
55
- llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
56
- --reranking --port 8090 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 512
57
-
58
- # === SOTA upgrade (16GB+ GPU) ===
7
+ ---
59
8
 
60
- # Embedding (zembed-1) -ub must match -b for non-causal attention
61
- llama-server -m zembed-1-Q4_K_M.gguf \
62
- --embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
9
+ ## Inference at a glance
63
10
 
64
- # Reranker (zerank-2) seq-cls SIDECAR (transformers, bf16), NOT a llama-server GGUF:
65
- # cd extras/rerankers/zerank-2-seq && docker compose run --rm convert && docker compose up -d reranker
66
- # The zerank-2 GGUF is deprecated: llama.cpp drops its score head -> near-zero, uninformative scores.
67
- ```
11
+ Three services**embedding**, **LLM** (query expansion / intent / A-MEM), **reranker**. Default: all three as `llama-server` with an in-process `node-llama-cpp` fallback that auto-downloads on first use (works with no GPU). The `bin/clawmem` wrapper points at `localhost:8088/8089/8090`. **Always run via `bin/clawmem`** — it sets the endpoints.
68
12
 
69
- ### Verify Endpoints
13
+ **Choose a stack:**
14
+ - **native** (default) — EmbeddingGemma-300M + qmd-query-expansion-1.7B + qwen3-reranker-0.6B · ~4 GB or in-process · **permissive, commercial OK** · zero-config.
15
+ - **z / SOTA** — zembed-1 + qmd-query-expansion-1.7B + zerank-2 seq-cls **sidecar** · ~16 GB · **CC-BY-NC-4.0, non-commercial only** · best recall.
16
+ - **cloud embedding** — Jina/OpenAI/Voyage/Cohere · embedding **only** (LLM + reranker stay local) · no local GPU needed.
70
17
 
71
- ```bash
72
- # Embedding
73
- curl http://host:8088/v1/embeddings -d '{"input":"test","model":"embedding"}' -H 'Content-Type: application/json'
18
+ **Landmines:**
19
+ - The zerank-2 **GGUF is inert** — llama.cpp drops the score head → ranking silently collapses to RRF. Use the **seq-cls sidecar**; verify with `clawmem rerank-health` (liveness ≠ correctness).
20
+ - `-ub` must equal `-b` for embedding/reranking (non-causal attention) or `llama-server` asserts.
21
+ - Changing embedding dimensions → `clawmem embed --force` (full re-embed).
22
+ - `CLAWMEM_NO_LOCAL_MODELS=true` to fail fast instead of silent CPU fallback.
74
23
 
75
- # LLM
76
- curl http://host:8089/v1/models
24
+ Stack decision matrix + server setup: [docs/guides/inference-services.md](docs/guides/inference-services.md) · cloud: [docs/guides/cloud-embedding.md](docs/guides/cloud-embedding.md) · all env vars: [docs/reference/configuration.md](docs/reference/configuration.md).
77
25
 
78
- # Reranker
79
- curl http://host:8090/v1/models
80
- ```
26
+ ---
81
27
 
82
- ## Environment Variable Reference
83
-
84
- | Variable | Default (via wrapper) | Effect |
85
- |---|---|---|
86
- | `CLAWMEM_EMBED_URL` | `http://localhost:8088` | Embedding server URL. Local llama-server, cloud API, or falls back to in-process `node-llama-cpp` if unset. |
87
- | `CLAWMEM_EMBED_API_KEY` | (none) | API key for cloud embedding providers. Sent as Bearer token. Enables cloud mode: skips client-side truncation, sends `truncate: true` + `task` param (LoRA adapter selection for Jina v5), and activates batch embedding with adaptive TPM-aware pacing. |
88
- | `CLAWMEM_EMBED_MODEL` | `embedding` | Model name for embedding requests. Override for cloud providers (e.g. `jina-embeddings-v5-text-small`). |
89
- | `CLAWMEM_EMBED_MAX_CHARS` | `6000` | Max chars per embedding input. Default fits EmbeddingGemma (2048 tokens). Set to `1100` for granite-278m (512 tokens). Cloud providers skip truncation. |
90
- | `CLAWMEM_EMBED_TPM_LIMIT` | `100000` | Tokens-per-minute limit for cloud embedding pacing. Match to your provider tier: Free 100000, Paid 2000000, Premium 50000000. |
91
- | `CLAWMEM_EMBED_DIMENSIONS` | (none) | Output dimensions for OpenAI `text-embedding-3-*` Matryoshka models (e.g. `512`, `1024`). Only sent when URL contains `openai.com`. |
92
- | `CLAWMEM_LLM_URL` | `http://localhost:8089` | LLM server for intent, expansion, A-MEM, and entity extraction. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. For better entity extraction quality, point at a 7B+ model or cloud API during `reindex --enrich` (see `docs/internals/entity-resolution.md`). |
93
- | `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. |
94
- | `CLAWMEM_NO_LOCAL_MODELS` | `false` | Blocks `node-llama-cpp` from auto-downloading GGUF models. Set `true` for remote-only setups. |
95
- | `CLAWMEM_VAULTS` | (none) | JSON map of vault name → SQLite path for multi-vault mode. E.g. `{"work":"~/.cache/clawmem/work.sqlite"}` |
96
- | `CLAWMEM_ENABLE_AMEM` | enabled | A-MEM note construction + link generation during indexing. |
97
- | `CLAWMEM_ENABLE_CONSOLIDATION` | disabled | Background worker backfills unenriched docs and runs Phase 2/3 consolidation + deductive synthesis. **v0.8.2:** every tick is wrapped in a DB-backed `worker_leases` row (`light-consolidation` key), so multiple host processes against the same vault cannot race on Phase 2 merge writes. Hosted by either `clawmem watch` (canonical, long-lived) or `clawmem mcp` (per-session fallback). |
98
- | `CLAWMEM_CONSOLIDATION_INTERVAL` | 300000 | Worker interval in ms (min 15000). |
99
- | `CLAWMEM_HEAVY_LANE` | disabled | **v0.8.0.** Enable the quiet-window heavy maintenance worker — a second, longer-interval consolidation lane with DB-backed `worker_leases` exclusivity, stale-first batching, and `maintenance_runs` journaling. Runs alongside the light lane; off by default. **v0.8.2:** canonical host is `clawmem watch` (e.g. systemd `clawmem-watcher.service`); `clawmem mcp` retains the same gate as a fallback host but emits a stderr warning advising operators to move heavy-lane hosting to the watcher because per-session stdio MCPs may never be alive during the configured quiet window. |
100
- | `CLAWMEM_HEAVY_LANE_INTERVAL` | 1800000 | **v0.8.0.** Heavy-lane tick interval in ms (min 30000, default 30 min). |
101
- | `CLAWMEM_HEAVY_LANE_WINDOW_START` | (none) | **v0.8.0.** Start hour (0-23) of the quiet window. Unset → no window. |
102
- | `CLAWMEM_HEAVY_LANE_WINDOW_END` | (none) | **v0.8.0.** End hour (0-23, exclusive) of the quiet window. Supports midnight wrap (22→6). |
103
- | `CLAWMEM_HEAVY_LANE_MAX_USAGES` | 30 | **v0.8.0.** Max `context_usage` rows in the last 10 min before the heavy lane skips with `reason='query_rate_high'`. |
104
- | `CLAWMEM_HEAVY_LANE_OBS_LIMIT` | 100 | **v0.8.0.** Phase 2 stale-first observation batch size. |
105
- | `CLAWMEM_HEAVY_LANE_DED_LIMIT` | 40 | **v0.8.0.** Phase 3 stale-first deductive candidate batch size. |
106
- | `CLAWMEM_HEAVY_LANE_SURPRISAL` | `false` | **v0.8.0.** When `true`, the heavy lane seeds Phase 2 with k-NN anomaly-ranked doc ids from `computeSurprisalScores` instead of stale-first ordering. Degrades to stale-first (`surprisal-fallback-stale` metric) on vaults without embeddings. |
107
- | `CLAWMEM_NUDGE_INTERVAL` | `15` | Prompts between lifecycle tool use before `<vault-nudge>` injection. 0 to disable. |
108
- | `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 merge-safety score threshold when candidate and existing anchors align. Merges above this normalized 3-gram cosine similarity are allowed. |
109
- | `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety score threshold (fallback when anchors are ambiguous). |
110
- | `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration before switching on the gate. |
111
- | `CLAWMEM_CONTRADICTION_POLICY` | `link` | **v0.7.1.** How the merge-time contradiction gate handles a contradictory merge. `link` keeps both rows active and inserts a `contradicts` edge. `supersede` marks the old row `status='inactive'`. |
112
- | `CLAWMEM_CONTRADICTION_MIN_CONFIDENCE` | `0.5` | **v0.7.1.** Minimum combined (heuristic + LLM) confidence required before the contradiction gate blocks a merge. Below this, the merge proceeds. |
113
-
114
- **Note:** The `bin/clawmem` wrapper sets all endpoint defaults. Always use the wrapper — never `bun run src/clawmem.ts` directly. For remote GPU setups, add the same env vars to the watcher service via a systemd drop-in.
115
-
116
- ## Quick Setup
28
+ ## Install
117
29
 
118
30
  ```bash
119
- # Install via npm
120
- bun add -g clawmem # or: npm install -g clawmem
121
-
122
- # Or from source
123
- git clone https://github.com/yoloshii/clawmem.git ~/clawmem
124
- cd ~/clawmem && bun install
125
- ln -sf ~/clawmem/bin/clawmem ~/.bun/bin/clawmem
126
-
127
- # Bootstrap a vault (init + index + embed + hooks + MCP)
128
- clawmem bootstrap ~/notes --name notes
31
+ npm install -g clawmem # or: bun add -g clawmem
32
+ clawmem bootstrap ~/notes --name notes # init + index + embed + hooks + MCP
129
33
 
130
34
  # Or step by step:
131
35
  clawmem init
132
36
  clawmem collection add ~/notes --name notes
133
37
  clawmem update --embed
134
- clawmem setup hooks
135
- clawmem setup mcp
136
-
137
- # Verify
138
- clawmem doctor # Full health check
139
- clawmem status # Quick index status
38
+ clawmem setup hooks && clawmem setup mcp
39
+ clawmem doctor # full health check (clawmem status = quick)
140
40
  ```
141
41
 
142
- ### Background Services (systemd user units)
143
-
144
- The watcher and embed timer keep the vault fresh automatically. Create these after setup:
145
-
146
- ```bash
147
- # Create systemd user directory
148
- mkdir -p ~/.config/systemd/user
149
-
150
- # clawmem-watcher.service — auto-indexes on .md changes
151
- cat > ~/.config/systemd/user/clawmem-watcher.service << 'EOF'
152
- [Unit]
153
- Description=ClawMem file watcher — auto-indexes on .md changes
154
- After=default.target
155
-
156
- [Service]
157
- Type=simple
158
- ExecStart=%h/clawmem/bin/clawmem watch
159
- Restart=on-failure
160
- RestartSec=10
161
-
162
- [Install]
163
- WantedBy=default.target
164
- EOF
165
-
166
- # clawmem-embed.service — oneshot embedding sweep
167
- cat > ~/.config/systemd/user/clawmem-embed.service << 'EOF'
168
- [Unit]
169
- Description=ClawMem embedding sweep
170
-
171
- [Service]
172
- Type=oneshot
173
- ExecStart=%h/clawmem/bin/clawmem embed
174
- EOF
175
-
176
- # clawmem-embed.timer — daily at 04:00
177
- cat > ~/.config/systemd/user/clawmem-embed.timer << 'EOF'
178
- [Unit]
179
- Description=ClawMem daily embedding sweep
180
-
181
- [Timer]
182
- OnCalendar=*-*-* 04:00:00
183
- Persistent=true
184
- RandomizedDelaySec=300
185
-
186
- [Install]
187
- WantedBy=timers.target
188
- EOF
189
-
190
- # Enable and start
191
- systemctl --user daemon-reload
192
- systemctl --user enable --now clawmem-watcher.service clawmem-embed.timer
193
-
194
- # Persist across reboots (start without login)
195
- loginctl enable-linger $(whoami)
196
-
197
- # Verify
198
- systemctl --user status clawmem-watcher.service clawmem-embed.timer
199
- ```
200
-
201
- **Note:** The service files use `%h` (home directory specifier). If clawmem is installed elsewhere, update `ExecStart` paths. For remote GPU setups, add `Environment=CLAWMEM_EMBED_URL=http://host:8088` etc. to both service files (the `bin/clawmem` wrapper sets defaults).
202
-
203
- ### Reranker health check (optional, recommended for remote-sidecar setups)
204
-
205
- `clawmem rerank-health` probes the reranker for **discrimination** (not just liveness) and exits non-zero when it is degenerate — catching the failure mode where a mis-converted reranker (e.g. the deprecated zerank-2 GGUF, whose llama.cpp conversion drops the score head) returns HTTP 200 + valid JSON but near-zero, non-discriminating scores, silently collapsing the final ranking to RRF. The same probe runs inside `clawmem doctor` (section 9); this scheduled unit alerts proactively without query traffic. Schedule it when the reranker is a remote sidecar that could be redeployed/reverted out from under you.
206
-
207
- ```bash
208
- # clawmem-rerank-health.service — oneshot probe; exits 1 if the reranker is degenerate
209
- cat > ~/.config/systemd/user/clawmem-rerank-health.service << 'EOF'
210
- [Unit]
211
- Description=ClawMem reranker discrimination health check
212
-
213
- [Service]
214
- Type=oneshot
215
- # Hard outer bound — a hung reranker must not hang the check (the probe also times out per-request).
216
- TimeoutStartSec=120
217
- ExecStart=%h/clawmem/bin/clawmem rerank-health
218
- # Alert on failure: point this at your notifier unit (ntfy, mail, etc.).
219
- OnFailure=clawmem-rerank-health-alert@%n.service
220
- EOF
221
-
222
- # clawmem-rerank-health.timer — every 6h (the failure it guards is rare + infra-driven)
223
- cat > ~/.config/systemd/user/clawmem-rerank-health.timer << 'EOF'
224
- [Unit]
225
- Description=ClawMem reranker health check (every 6h)
226
-
227
- [Timer]
228
- OnCalendar=*-*-* 00/6:00:00
229
- Persistent=true
230
- RandomizedDelaySec=300
231
-
232
- [Install]
233
- WantedBy=timers.target
234
- EOF
235
-
236
- systemctl --user daemon-reload
237
- systemctl --user enable --now clawmem-rerank-health.timer
238
- ```
239
-
240
- For remote GPU setups, add `Environment=CLAWMEM_RERANK_URL=http://host:8090` (+ embed/LLM) to the `.service`. `OnFailure=` is the primary alert path; the `curator-nudge` SessionStart hook is a secondary "you missed the page" surface. Run `clawmem rerank-health` (or `--json`) manually any time to check on demand.
241
-
242
- ---
243
-
244
- ## OpenClaw Integration: Memory System Configuration
245
-
246
- When using ClawMem with OpenClaw, choose one of two deployment options:
247
-
248
- **Active Memory coexistence:** ClawMem is fully compatible with OpenClaw's Active Memory plugin (v2026.4.10+). They search different backends (ClawMem vault vs dreaming/wiki) and inject into different prompt regions (user prompt vs system prompt). Both can run simultaneously — no configuration needed.
249
-
250
- **memory-core dreaming sidecar coexistence (v2026.4.18+, #65411):** ClawMem and `memory-core` dreaming can also run side-by-side. When ClawMem owns the memory slot AND `plugins.entries.memory-core.config.dreaming.enabled = true`, OpenClaw loads `memory-core`'s dreaming engine alongside ClawMem (rest of `memory-core` stays unloaded). Dreaming continues writing to `memory/dreaming/{phase}/YYYY-MM-DD.md` (separate from ClawMem's vault). Default after `openclaw plugins enable clawmem` is `dreaming.enabled = false` (ClawMem-only). Set it `true` if you want to keep the dreaming output stream alongside ClawMem.
251
-
252
- **OpenClaw v2026.4.11+ required for ClawMem v0.10.0+.** v2026.4.11 tightened plugin discovery (`readdirSync({ withFileTypes: true })` + `dirent.isDirectory()`) and added a plugin-directory ownership check. ClawMem v0.10.0+ ships the new discovery manifest (`src/openclaw/package.json` with `openclaw.extensions`) and defaults to recursive copy (not symlink) installs to clear both gates. Older notes: v2026.4.10 fixed #64192 (`plugins.slots.contextEngine` silently dropped during config normalization, relevant only for pre-v0.10.0 ClawMem that still used the `contextEngine` slot).
253
-
254
- **ClawMem v0.10.0 uses the `memory` slot, not `contextEngine`.** `openclaw plugins enable clawmem` sets `plugins.slots.memory: "clawmem"` and disables competing memory plugins (`memory-core`, `memory-lancedb`) in one step. The older `openclaw config set plugins.slots.contextEngine clawmem` pattern does not apply to v0.10.0+.
255
-
256
- ### Option 1: ClawMem Exclusive (Recommended)
257
-
258
- ClawMem handles 100% of structured memory. Disable native memory search (not Active Memory — that's separate and compatible):
259
-
260
- **Configuration:**
261
- ```bash
262
- # Disable OpenClaw's native memory search
263
- openclaw config set agents.defaults.memorySearch.extraPaths "[]"
264
-
265
- # Verify
266
- openclaw config get agents.defaults.memorySearch
267
- # Expected: {"extraPaths": []}
268
-
269
- # Confirm no native memory index exists
270
- ls ~/.openclaw/agents/main/memory/
271
- # Expected: "No such file or directory"
272
- ```
273
-
274
- **Memory distribution:**
275
- - **Tier 2 (90%):** Hooks auto-inject context (session-bootstrap, context-surfacing, staleness-check, decision-extractor, handoff-generator, feedback-loop)
276
- - **Tier 3 (10%):** Agent-initiated MCP tools (query, intent_search, find_causal_links, etc.)
277
-
278
- ### Option 2: Hybrid (ClawMem + Native)
279
-
280
- Run both ClawMem and OpenClaw's native memory search for redundancy.
281
-
282
- **Configuration:**
283
- ```bash
284
- openclaw config set agents.defaults.memorySearch.extraPaths '["~/documents", "~/notes"]'
285
- ```
286
-
287
- **Tradeoffs:**
288
- - Redundant recall from two independent systems
289
- - 10-15% context window waste from duplicate facts
290
- - Two memory indices to maintain
291
-
292
- **Recommendation:** Use Option 1 unless you have a specific need for redundant memory systems.
42
+ Quickstart + index patterns: [docs/quickstart.md](docs/quickstart.md) · hooks/MCP: [docs/guides/setup-hooks.md](docs/guides/setup-hooks.md), [docs/guides/setup-mcp.md](docs/guides/setup-mcp.md) · background services: [docs/guides/systemd-services.md](docs/guides/systemd-services.md) · upgrading: [docs/guides/upgrading.md](docs/guides/upgrading.md).
293
43
 
294
44
  ---
295
45
 
296
- ## Memory Retrieval (90/10 Rule)
297
-
298
- ClawMem hooks handle ~90% of retrieval automatically. Agent-initiated MCP calls cover the remaining ~10%.
299
-
300
- ### Tier 2 — Automatic (hooks, zero agent effort)
46
+ ## Memory retrieval (90/10 rule)
301
47
 
302
- | Hook | Trigger | Budget | Content |
303
- |------|---------|--------|---------|
304
- | `context-surfacing` | UserPromptSubmit | profile-driven (default 800 + factsTokens sub-budget) | retrieval gate → **multi-turn query construction** (v0.8.1: current prompt + up to 2 recent same-session priors from `context_usage.query_text`, 10-min max age, capped at 2000 chars with current-first preservation — used only for discovery: vector/FTS/expansion, NOT for rerank/scoring/snippet extraction) → **session focus topic resolution** (v0.9.0 §11.4: reads per-session focus file at `~/.cache/clawmem/sessions/<id>.focus`, threaded as intent hint to `expandQuery` + `rerank` + `extractSnippet`) → profile-driven hybrid search (vector if `useVector`, timeout from profile) → FTS supplement → file-aware supplemental search (E13, raw current prompt) → snooze filter → noise filter → spreading activation (E11: co-activated doc boost) → composite scoring → **session focus topic boost** (v0.9.0 §11.4: 1.4× match / 0.75× demote floor 50%, NO-OP on zero matches to preserve baseline ordering) → adaptive threshold → memory type diversification (E10) → tiered injection (HOT/WARM/COLD snippets) → `<vault-context><instruction>…</instruction><facts>…</facts><relationships>…</relationships><vault-facts>…</vault-facts></vault-context>` (v0.7.1: instruction always prepended when context is returned; relationships block lists memory-graph edges where BOTH endpoints are in the surfaced set, truncated first when over budget. **v0.9.0 §11.1:** `<vault-facts>` KG injection block appends raw SPO triple lines from entities seeded by the prompt via three-path prompt-only extraction — canonical IDs + proper nouns + longer-first n-grams — with a dedicated `factsTokens` sub-budget per profile (speed=0 disables the stage, balanced=200, deep=250), cross-entity triple dedup, and truncate-at-triple-boundary budget discipline; fail-open on every error path) + optional `<vault-routing>` hint. Budget, max results, vector timeout, min score, and facts sub-budget all driven by `CLAWMEM_PROFILE`. Raw prompt persisted to `context_usage.query_text` for future multi-turn lookback — except on gated skip paths (slash commands, heartbeats, too-short prompts) where the text is withheld for privacy. |
305
- | `postcompact-inject` | SessionStart (compact) | 1200 tokens | re-injects authoritative context after compaction: precompact state (600) + recent decisions (400) + antipatterns (150) + vault context (200) → `<vault-postcompact>` |
306
- | `curator-nudge` | SessionStart | 200 tokens | surfaces curator report actions, nudges when report is stale (>7 days) |
307
- | `precompact-extract` | PreCompact | — | extracts decisions, file paths, open questions → writes `precompact-state.md` to auto-memory. Query-aware decision ranking. Reindexes auto-memory collection. |
308
- | `decision-extractor` | Stop | — | LLM extracts observations → `_clawmem/agent/observations/`, infers causal links, detects contradictions, persists observer-emitted SPO triples via `ensureEntityCanonical` (canonical `vault:type:slug` IDs shared with A-MEM) using the tight predicate vocabulary (adopted, migrated_to, deployed_to, runs_on, replaced, depends_on, integrates_with, uses, prefers, avoids, caused_by, resolved_by, owned_by). Eligible observation types: decision/preference/milestone/problem/discovery/feature. Background consolidation worker synthesizes deductive observations from related facts (Phase 3, every ~15 min). |
309
- | `handoff-generator` | Stop | — | LLM summarizes session → `_clawmem/agent/handoffs/` |
310
- | `feedback-loop` | Stop | — | tracks referenced notes → boosts confidence, records usage relations + co-activations between co-referenced docs, tracks utility signals (surfaced vs referenced ratio for lifecycle automation), per-turn recall attribution (marks which surfaced docs were cited in which turn) |
48
+ ### Tier 2 automatic (hooks, zero agent effort)
311
49
 
312
- **Default behavior:** Read injected `<vault-context>` first. If sufficient, answer immediately.
50
+ | Hook | Trigger | Does |
51
+ |------|---------|------|
52
+ | `context-surfacing` | UserPromptSubmit | retrieval gate → profile-driven hybrid search → FTS supplement → file-aware search → snooze/noise filters → spreading activation → memory-type diversification → tiered injection → `<vault-context>` (+ optional `<vault-facts>` SPO triples, `<vault-routing>` hint). Budget/results/timeout/threshold driven by `CLAWMEM_PROFILE`. |
53
+ | `postcompact-inject` | SessionStart (compact) | re-injects authoritative state after compaction → `<vault-postcompact>` |
54
+ | `curator-nudge` | SessionStart | surfaces curator actions; nudges when the report is stale |
55
+ | `precompact-extract` | PreCompact | extracts decisions / file paths / open questions before compaction |
56
+ | `decision-extractor` | Stop | LLM → observations + causal links + contradiction detection + SPO triples |
57
+ | `handoff-generator` | Stop | LLM session summary → handoffs |
58
+ | `feedback-loop` | Stop | tracks referenced notes → confidence boosts, co-activations, utility signals |
313
59
 
314
- **Hook blind spots (by design):** Hooks filter out `_clawmem/` system artifacts, enforce score thresholds, and cap token budget. Absence in `<vault-context>` does NOT mean absence in memory. If you expect a memory to exist but it wasn't surfaced, escalate to Tier 3.
60
+ **Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
61
+ **Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
62
+ **Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) + an activation floor; only `deep` adds query expansion + reranking to the hook path. → [docs/concepts/hooks-vs-mcp.md](docs/concepts/hooks-vs-mcp.md).
315
63
 
316
- **Adaptive thresholds:** Context-surfacing uses ratio-based scoring that adapts to vault characteristics (size, document quality, content age, embedding model). Results are kept within a percentage of the best result's composite score rather than a fixed absolute threshold. An activation floor prevents surfacing when all results are weak. Profiles control the ratio: `speed` (65%), `balanced` (55%), `deep` (45% + query expansion + reranking). `CLAWMEM_PROFILE=deep` is recommended for vaults with older content or lower-quality documents. MCP tools use fixed absolute thresholds, not adaptive.
64
+ ### Tier 3 agent-initiated (one targeted MCP call)
317
65
 
318
- ### Tier 3Agent-Initiated (one targeted MCP call)
66
+ **3-rule escalation gateescalate ONLY when one fires:**
67
+ 1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact the task requires.
68
+ 2. **Cross-session question** — "why did we decide X", "what changed since last time", "when did we start Y".
69
+ 3. **Pre-irreversible check** — before a destructive / hard-to-reverse change, check the vault for prior decisions.
319
70
 
320
- **Escalate ONLY when one of these three rules fires:**
321
- 1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact/chain the task requires. Hooks surface top-k by relevance; if the needed memory wasn't in top-k, escalate.
322
- 2. **Cross-session question** — the task explicitly references prior sessions or decisions: "why did we decide X", "what changed since last time", "when did we start doing Y".
323
- 3. **Pre-irreversible check** — about to make a destructive or hard-to-reverse change (deletion, config change, architecture decision). Check vault for prior decisions before proceeding.
71
+ All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculatively.**
324
72
 
325
- All other retrieval is handled by Tier 2 hooks. Do NOT call MCP tools speculatively or "just to be thorough."
326
-
327
- **Once escalated, route by query type:**
328
-
329
- **PREFERRED:** `memory_retrieve(query)` — auto-classifies and routes to the optimal backend (query, intent_search, session_log, find_similar, or query_plan). Use this instead of manually choosing a tool below.
73
+ **Routing** PREFERRED: `memory_retrieve(query)` auto-classifies and routes to the optimal backend.
330
74
 
331
75
  ```
332
- 1a. General recall query(query, compact=true, limit=20)
333
- Full hybrid: BM25 + vector + query expansion + deep reranking (4000 char).
334
- Supports compact, collection filter (comma-separated for multi-collection: `"col1,col2"`), intent, and candidateLimit.
335
- Default for most Tier 3 needs.
336
- Optional: intent="domain hint" for ambiguous queries (steers expansion, reranking, chunk selection, snippets).
337
- Optional: candidateLimit=N to tune precision/speed (default 30).
338
- BM25 strong-signal bypass: skips expansion when top BM25 hit ≥ 0.85 with gap ≥ 0.15 (disabled when intent is provided).
339
-
340
- 1b. Causal/why/when/entity → intent_search(query, enable_graph_traversal=true)
76
+ 1a. General recall -> query(query, compact=true, limit=20)
77
+ Full hybrid: BM25 + vector + expansion + deep rerank. Supports compact, collection,
78
+ intent, candidateLimit. BM25 strong-signal bypass skips expansion when top hit >= 0.85
79
+ with gap >= 0.15 (disabled when intent is provided).
80
+ 1b. Causal/why/when/entity -> intent_search(query, enable_graph_traversal=true)
341
81
  MAGMA intent classification + intent-weighted RRF + multi-hop graph traversal.
342
- Use DIRECTLY (not as fallback) when the question is "why", "when", "how did X lead to Y",
343
- or needs entity-relationship traversal.
344
- Override auto-detection: force_intent="WHY"|"WHEN"|"ENTITY"|"WHAT"
345
- When to override:
346
- WHY "why", "what led to", "rationale", "tradeoff", "decision behind"
347
- ENTITY named component/person/service needing cross-doc linkage, not just keyword hits
348
- WHEN timelines, first/last occurrence, "when did this change/regress"
349
- WHEN note: start with enable_graph_traversal=false (BM25-biased); fall back to query() if recall drifts.
350
-
351
- Choose 1a or 1b based on query type. They are parallel options, not sequential.
352
-
353
- 1c. Multi-topic/complex query_plan(query, compact=true)
354
- Decomposes query into 2-4 typed clauses (bm25/vector/graph), executes in parallel, merges via RRF.
355
- Use when query spans multiple topics or needs both keyword and semantic recall simultaneously.
356
- Falls back to single-query behavior for simple queries (planner returns 1 clause).
82
+ Use DIRECTLY (not as a fallback) for "why" / "when" / "how did X lead to Y" / entity links.
83
+ Override: force_intent="WHY"|"WHEN"|"ENTITY"|"WHAT". (1a vs 1b are parallel, chosen by query type.)
84
+ 1c. Multi-topic -> query_plan(query, compact=true)
85
+ Decomposes into 2-4 typed clauses (bm25/vector/graph), runs them in parallel, merges via RRF.
86
+ 2. Progressive disclosure -> multi_get("path1,path2") for full content of top hits
87
+ 3. Spot checks -> search(query) (BM25, 0 GPU) or vsearch(query) (vector, 1 GPU)
88
+ 4. Chain tracing -> find_causal_links(docid, direction="both", depth=5)
89
+ 5. Entity facts -> kg_query(entity) (SPO triples; NOT causal "why" that's intent_search)
90
+ 6. Temporal context -> timeline(docid, before=5, after=5)
91
+ ```
92
+
93
+ | Tool | Purpose |
94
+ |------|---------|
95
+ | `memory_retrieve` | **Preferred.** Auto-classifies + routes. Use instead of choosing manually. |
96
+ | `query` | Full hybrid (BM25 + vector + rerank). General-purpose. WRONG for "why" ( `intent_search`) or cross-session (→ `session_log`). |
97
+ | `intent_search` | "why did we decide X" / "what caused Y" / "who worked on Z". Classifies intent, traverses graph edges — decision chains `query` can't find. |
98
+ | `query_plan` | Multi-topic ("X and also Y", "compare A with B"). Splits + routes each clause. |
99
+ | `search` | BM25 keyword — exact terms, config names, error codes. Fast, 0 GPU. |
100
+ | `vsearch` | Vector semantic — conceptual/fuzzy when vocabulary unknown. ~100ms, 1 GPU. |
101
+ | `get` / `multi_get` | Single doc by path/`#docid` / multiple by glob or comma-list. |
102
+ | `find_similar` | "what else relates to X" — k-NN vector neighbors beyond keyword overlap. |
103
+ | `find_causal_links` | Trace decision chains ("what led to X") over observation docs. |
104
+ | `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why". |
105
+ | `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
106
+ | `profile` | User profile (static facts + dynamic context). |
107
+ | `memory_pin` | +0.3 composite boost. PROACTIVELY for constraints, architecture decisions, corrections. |
108
+ | `memory_snooze` | PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
109
+ | `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
110
+ | `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
111
+ | `timeline` | Temporal neighborhood around a doc. Progressive disclosure: search → timeline → get. |
112
+ | `memory_evolution_status` | How a doc's A-MEM metadata evolved over time. |
113
+ | `lifecycle_status` / `lifecycle_sweep` / `lifecycle_restore` | Lifecycle stats / archive stale (dry-run default) / restore. |
114
+ | `index_stats` / `status` / `reindex` | Doc counts + embedding coverage / quick health / force re-index (does NOT embed). |
115
+ | `beads_sync` / `vault_sync` / `list_vaults` | Beads from Dolt / index a dir into a named vault / list vaults. |
116
+ | `diary_write` / `diary_read` | Diary entries — non-hooked envs only (in Claude Code hooks capture this). |
117
+
118
+ **Multi-vault:** all tools accept an optional `vault` param (omit for single-vault mode). **Progressive disclosure:** ALWAYS `compact=true` first → review snippets/scores → `get` / `multi_get` for full content. → full param docs: [docs/reference/mcp-tools.md](docs/reference/mcp-tools.md).
357
119
 
358
- 2. Progressive disclosure → multi_get("path1,path2") for full content of top hits
359
-
360
- 3. Spot checks → search(query) (BM25, 0 GPU) or vsearch(query) (vector, 1 GPU)
361
-
362
- 4. Chain tracing → find_causal_links(docid, direction="both", depth=5)
363
- Traverses causal edges between _clawmem/agent/observations/ docs (from decision-extractor).
364
-
365
- 5. Entity facts → kg_query(entity, as_of?, direction?)
366
- Structured SPO triples with temporal validity. Different from intent_search:
367
- - kg_query: "what does ClawMem relate to?" → returns structured facts (subject-predicate-object)
368
- - intent_search: "why did we choose ClawMem?" → returns documents with causal reasoning
369
- Use kg_query for entity lookup, intent_search for causal chains.
370
-
371
- 6. Memory debugging → memory_evolution_status(docid)
372
-
373
- 7. Temporal context → timeline(docid, before=5, after=5, same_collection=false)
374
- Shows what was created/modified before and after a document.
375
- Use after search to understand chronological neighborhood.
376
- ```
377
-
378
- **Other tools:**
379
- - `find_similar(docid)` — "what else relates to X". k-NN vector neighbors — discovers connections beyond keyword overlap.
380
- - `session_log` — USE THIS for "last time", "yesterday", "what did we do". DO NOT use `query()` for cross-session questions.
381
- - `profile` — user profile (static facts + dynamic context).
382
- - `memory_forget(query)` — deactivate a memory by closest match.
383
- - `memory_pin(query, unpin?)` — +0.3 composite boost. USE PROACTIVELY for constraints, architecture decisions, corrections.
384
- - `memory_snooze(query, until?)` — USE PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days.
385
- - `build_graphs(temporal?, semantic?)` — build temporal backbone + semantic graph after bulk ingestion. Not needed after routine indexing (A-MEM handles per-doc links).
386
- - `beads_sync(project_path?)` — sync Beads issues from Dolt backend (via `bd` CLI) into memory. Usually automatic via watcher.
387
- - `query_plan(query, compact=true)` — USE THIS for multi-topic queries. `query()` searches as one blob — this splits topics and routes each optimally.
388
- - `timeline(docid, before=5, after=5, same_collection=false)` — temporal neighborhood around a document. Progressive disclosure: search → timeline → get. Supports same-collection scoping and session correlation.
389
- - `list_vaults()` — show configured vault names and paths. Empty in single-vault mode (default).
390
- - `vault_sync(vault, content_root, pattern?, collection_name?)` — index markdown from a directory into a named vault. Restricted-path validation rejects sensitive directories (`/etc/`, `/root/`, `.ssh`, `.env`, `credentials`, etc.).
391
- - `kg_query(entity, as_of?, direction?)` — query the SPO knowledge graph for an entity's relationships. Returns temporal triples with validity windows. USE THIS for "what does X relate to?", "what was true about X in January?". Uses entity resolution for lookup.
392
- - `diary_write(entry, topic?, agent?)` — write a diary entry. USE PROACTIVELY in non-hooked environments (Hermes, Gemini, plain MCP) for recording important events and decisions. Do NOT use in Claude Code (hooks handle this automatically).
393
- - `diary_read(last_n?, agent?)` — read recent diary entries.
394
-
395
- ### Multi-Vault
396
-
397
- All tools accept an optional `vault` parameter — retrieval (query, search, vsearch, intent_search, memory_retrieve, query_plan), document access (get, multi_get, find_similar, find_causal_links, timeline, memory_evolution_status, session_log), mutations (memory_pin, memory_snooze, memory_forget), lifecycle (lifecycle_status, lifecycle_sweep, lifecycle_restore), and maintenance (status, reindex, index_stats, build_graphs). Omit `vault` for the default vault (single-vault mode). Named vaults are configured in `~/.config/clawmem/config.yaml` under `vaults:` or via `CLAWMEM_VAULTS` env var. Vault paths support `~` expansion.
398
-
399
- ### Memory Lifecycle
400
-
401
- Pin, snooze, and forget are **manual MCP tools** — not automated. The agent should proactively use them when appropriate:
402
-
403
- - **Pin** (`memory_pin`) — +0.3 composite boost, ensures persistent surfacing.
404
- - **Proactive triggers:** User says "remember this" / "don't forget" / "this is important". Architecture or critical design decision just made. User-stated preference or constraint that should persist across sessions.
405
- - **Do NOT pin:** routine decisions, session-specific context, or observations that will naturally surface via recency.
406
- - **Snooze** (`memory_snooze`) — temporarily hides from context surfacing until a date.
407
- - **Proactive triggers:** A memory keeps surfacing but isn't relevant to current work. User says "not now" / "later" / "ignore this for now". Seasonal or time-boxed content (e.g., "revisit after launch").
408
- - **Forget** (`memory_forget`) — permanently deactivates. Use sparingly.
409
- - Only when a memory is genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
410
- - **Contradictions auto-resolve:** When `decision-extractor` detects a new decision contradicting an old one, the old decision's confidence is lowered automatically. No manual intervention needed for superseded decisions. **v0.7.1:** the consolidation worker adds a merge-time contradiction gate — before any Phase 2 merge, it runs a deterministic heuristic + LLM check and either links contradictory observations via a `contradicts` edge (default) or marks the prior row `status='inactive'` (when `CLAWMEM_CONTRADICTION_POLICY=supersede`). Phase 3 deductive synthesis applies the same gate to deductive dedupe matches.
411
-
412
- ### Anti-Patterns
413
-
414
- - Do NOT manually pick query/intent_search/search when `memory_retrieve` can auto-route.
415
- - Do NOT call MCP tools every turn — three rules above are the only gates.
416
- - Do NOT re-search what's already in `<vault-context>`.
417
- - Do NOT run `status` routinely. Only when retrieval feels broken or after large ingestion.
418
- - Do NOT pin everything — pin is for persistent high-priority items, not temporary boosting.
419
- - Do NOT forget memories to "clean up" — let confidence decay and contradiction detection handle it naturally.
420
- - Do NOT run `build_graphs` after every reindex — A-MEM creates per-doc links automatically. Only after bulk ingestion or when `intent_search` returns weak graph results.
421
- - Do NOT run `clawmem mine` autonomously — it is a bulk ingestion command (same category as `update`/`reindex`). Suggest it to the user when they mention old conversation exports, but let them run it. Bulk import has disk/embedding cost implications that need user consent. **v0.7.2 adds `--synthesize`** — an opt-in post-import LLM fact extraction pass that turns raw conversation dumps into searchable structured decisions / preferences / milestones / problems with cross-fact relations. Off by default; also requires user consent because it drives additional LLM calls (one per conversation doc). Suggest both together when the user wants to get real value out of old chat exports, not just the raw dumps.
422
- - Do NOT use `diary_write` in Claude Code — hooks (`decision-extractor`, `handoff-generator`) capture this automatically. Diary is for non-hooked environments only (Hermes, Gemini, plain MCP clients).
423
- - Do NOT use `kg_query` for causal "why" questions — use `intent_search` or `memory_retrieve`. `kg_query` returns structured entity facts (SPO triples), not reasoning chains.
424
-
425
- ## Tool Selection (one-liner)
426
-
427
- ```
428
- ClawMem escalation: memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) → multi_get → search/vsearch (spot checks)
429
- ```
430
-
431
- ## Curator Agent
432
-
433
- Maintenance agent for Tier 3 operations the main agent typically neglects. Install with `clawmem setup curator`.
434
-
435
- **Invoke:** "curate memory", "run curator", or "memory maintenance"
436
-
437
- **6 phases:**
438
- 1. Health snapshot — status, index_stats, lifecycle_status, doctor
439
- 2. Lifecycle triage — pin high-value unpinned memories, snooze stale content, propose forget candidates (never auto-confirms)
440
- 3. Retrieval health check — 5 probes (BM25, vector, hybrid, intent/graph, lifecycle)
441
- 4. Maintenance — reflect (cross-session patterns), consolidate --dry-run (dedup candidates)
442
- 5. Graph rebuild — conditional on probe results and embedding state
443
- 6. Collection hygiene — orphan detection, content type distribution
444
-
445
- **Safety rails:** Never auto-confirms forget. Never runs embed (timer's job). Never modifies config.yaml. All destructive proposals require user approval.
446
-
447
- ## Query Optimization
448
-
449
- The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector via RRF, and reranks with a cross-encoder. Agents do NOT choose search types — the pipeline handles fusion internally. The optimization levers are: **tool selection**, **query string quality**, **intent**, and **candidateLimit**.
450
-
451
- ### Tool Selection (highest impact)
452
-
453
- Pick the lightest tool that satisfies the need:
454
-
455
- | Tool | Cost | When |
456
- |------|------|------|
457
- | `search(q, compact=true)` | BM25 only, 0 GPU | Know exact terms, spot-check, fast keyword lookup |
458
- | `vsearch(q, compact=true)` | Vector only, 1 GPU call | Conceptual/fuzzy, don't know vocabulary |
459
- | `query(q, compact=true)` | Full hybrid, 3+ GPU calls | General recall, unsure which signal matters |
460
- | `intent_search(q)` | Hybrid + graph | Why/entity chains (graph traversal), when queries (BM25-biased) |
461
- | `query_plan(q)` | Hybrid + decomposition | Complex multi-topic queries needing parallel typed retrieval |
462
-
463
- Use `search` for quick keyword spot-checks. Use `query` for general recall (default Tier 3 workhorse). Use `intent_search` directly (not as fallback) when the question is causal or relational.
464
-
465
- ### Query String Quality
466
-
467
- The query string feeds BM25 directly (probes first, can short-circuit the pipeline) and anchors the 2×-weighted original signal in RRF.
468
-
469
- **For keyword recall (BM25):** 2-5 precise terms, no filler. Code identifiers work. BM25 AND's all terms as prefix matches (`perf` matches "performance") — no phrase search or negation syntax. A strong hit (≥ 0.85 with gap ≥ 0.15) skips expansion — faster results.
470
-
471
- **For semantic recall (vector):** Full natural language question, be specific. `"in the payment service, how are refunds processed"` > `"refunds"`.
472
-
473
- **Do NOT write hypothetical-answer-style queries.** The expansion LLM already generates hyde variants internally. A long hypothetical dilutes BM25 scoring and duplicates what the pipeline does autonomously.
474
-
475
- ### Intent Parameter
120
+ ---
476
121
 
477
- Steers 5 autonomous stages: expansion, reranking, chunk selection, snippet extraction, and strong-signal bypass (disabled when intent is provided).
122
+ ## Query optimization (4 levers)
478
123
 
479
- Use when: query term has multiple meanings in the vault, domain is known but query alone is ambiguous.
480
- Do NOT use when: query is already specific, single-domain vault, using `search`/`vsearch` (intent only affects `query`).
124
+ The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector via RRF, and reranks with a cross-encoder you do NOT choose search types. Your levers:
481
125
 
482
- Note: intent disables BM25 strong-signal bypass, forcing full expansion+reranking. Correct behavior intent means the query is ambiguous, so keyword confidence alone is insufficient.
126
+ 1. **Tool selection (highest impact)** — pick the lightest tool: `search` (BM25, 0 GPU, exact terms) < `vsearch` (vector, 1 GPU, fuzzy) < `query` (full hybrid) < `intent_search` (why/entity chains) < `query_plan` (multi-topic).
127
+ 2. **Query string quality** — keyword recall: 2–5 precise terms, no filler (BM25 ANDs prefix matches, no phrase/negation). Semantic recall: full natural-language question. Do NOT write hypothetical-answer queries (the LLM already generates hyde variants).
128
+ 3. **Intent (disambiguation)** — `query("performance", intent="web page load times")` steers 5 stages; use when the term is polysemous or the domain is ambiguous; skip when already specific. Disables BM25 strong-signal bypass (correct — intent signals ambiguity).
129
+ 4. **candidateLimit** — RRF candidates reaching the reranker (default 30). Lower (15) for speed/small vault; higher (50) for broad topics/recall.
483
130
 
484
- ## Composite Scoring (automatic, applied to all search tools)
131
+ pipeline internals: [docs/internals/query-pipeline.md](docs/internals/query-pipeline.md), [docs/internals/intent-search-pipeline.md](docs/internals/intent-search-pipeline.md). Deeper operating guidance: `SKILL.md`.
485
132
 
486
- **Observation invalidation (v0.2.0):** Documents with `invalidated_at` set are excluded from search results. Soft invalidation uses `invalidated_at`, `invalidated_by`, and `superseded_by` columns. When `decision-extractor` detects a contradiction that drops confidence ≤ 0.2, the observation is marked invalidated. Invalidated docs can be restored via lifecycle tools.
133
+ ---
487
134
 
488
- ```
489
- compositeScore = (0.50 × searchScore + 0.25 × recencyScore + 0.25 × confidenceScore) × qualityMultiplier × coActivationBoost
490
- ```
135
+ ## Composite scoring
491
136
 
492
- Where `qualityMultiplier = 0.7 + 0.6 × qualityScore` (range: 0.7× penalty to 1.3× boost).
493
- `coActivationBoost = 1 + min(coCount/10, 0.15)` — documents frequently surfaced together get up to 15% boost.
494
- Length normalization: `1/(1 + 0.5 × log2(max(bodyLength/500, 1)))` — penalizes verbose entries, floor at 30%.
495
- Frequency boost: `freqSignal = (revisions-1)×2 + (duplicates-1)`, `freqBoost = min(0.10, log1p(freqSignal)×0.03)`. Revision count weighted 2× vs duplicate count. Capped at 10%.
496
- Pinned documents get +0.3 additive boost (capped at 1.0).
137
+ Applied automatically to all search results.
497
138
 
498
- Recency intent detected ("latest", "recent", "last session"):
499
139
  ```
500
- compositeScore = (0.10 × searchScore + 0.70 × recencyScore + 0.20 × confidenceScore) × qualityMultiplier × coActivationBoost
140
+ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
501
141
  ```
502
142
 
503
- **`query` tool (v0.13.0+):** for non-recency queries the `query` tool uses retrieval-tuned weights `0.70 × searchScore + 0.15 × recencyScore + 0.15 × confidenceScore` (from a held-out judged eval). Recency intent still switches to the `0.10 / 0.70 / 0.20` weights above. `search`, `vsearch`, `memory_retrieve`, and the context-surfacing hook keep the `0.50 / 0.25 / 0.25` default.
504
-
505
- | Content Type | Half-Life | Effect |
506
- |--------------|-----------|--------|
507
- | decision, deductive, preference, hub | ∞ | Never decay |
508
- | antipattern | ∞ | Never decay — accumulated negative patterns persist |
509
- | project | 120 days | Slow decay |
510
- | research | 90 days | Moderate decay |
511
- | problem, milestone, note | 60 days | Default |
512
- | conversation, progress | 45 days | Faster decay |
513
- | handoff | 30 days | Fast — recent matters most |
143
+ - `qualityMultiplier = 0.7 + 0.6·qualityScore` (0. 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive**.
144
+ - **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/context-surfacing keep the 0.50/0.25/0.25 default.
145
+ - **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
146
+ - **Half-lives:** decision/deductive/preference/hub/antipattern = ∞ · project 120d · research 90d · problem/milestone/note 60d · conversation/progress 45d · handoff 30d (extend up to 3× for frequently-accessed).
514
147
 
515
- Half-lives extend up to 3× for frequently-accessed memories (access reinforcement decays over 90 days).
516
- Attention decay: non-durable types (handoff, progress, conversation, note, project) lose 5% confidence per week without access. Decision/deductive/preference/hub/research/antipattern are exempt.
148
+ full derivation: [docs/concepts/composite-scoring.md](docs/concepts/composite-scoring.md).
517
149
 
518
- ## Indexing & Graph Building
519
-
520
- ### What Gets Indexed (per collection in config.yaml, symlinked as index.yml)
521
-
522
- - `**/MEMORY.md` — any depth
523
- - `**/memory/**/*.md`, `**/memory/**/*.txt` — session logs
524
- - `**/docs/**/*.md`, `**/docs/**/*.txt` — documentation
525
- - `**/research/**/*.md`, `**/research/**/*.txt` — research dumps
526
- - `**/YYYY-MM-DD*.md`, `**/YYYY-MM-DD*.txt` — date-format records
527
-
528
- ### Excluded (even if pattern matches)
529
-
530
- - `gits/`, `scraped/`, `.git/`, `node_modules/`, `dist/`, `build/`, `vendor/`
531
-
532
- ### Indexing vs Embedding (important distinction)
533
-
534
- **Infrastructure (Tier 1, no agent action needed):**
535
- - **`clawmem-watcher`** — keeps index + A-MEM fresh (continuous, on `.md` change). Also watches `.beads/` — routes changes to `syncBeadsIssues()` which queries `bd` CLI for live Dolt data (auto-bridges deps into `memory_relations`). Does NOT embed.
536
- - **`clawmem-embed` timer** — keeps embeddings fresh (daily). Idempotent, skips already-embedded fragments.
537
-
538
- **Quality scoring:** Each document gets a `quality_score` (0.0–1.0) computed during indexing based on length, structure (headings, lists), decision keywords, correction keywords, and frontmatter richness. Applied as a multiplier in composite scoring.
539
-
540
- **Impact of missing embeddings:** `vsearch`, `query` (vector component), `context-surfacing` (vector component), and `generateMemoryLinks()` (neighbor discovery) all depend on embeddings. If embeddings are missing, these degrade silently — BM25 still works, but vector recall and inter-doc link quality suffer.
541
-
542
- **Agent escape hatches (rare):**
543
- - `clawmem embed` via CLI if you just wrote a doc and need immediate vector recall in the next turn.
544
- - Manual `reindex` only when immediate index freshness is required and watcher hasn't caught up.
545
-
546
- ### Graph Population (memory_relations)
547
-
548
- The `memory_relations` table is populated by multiple independent sources:
150
+ ---
549
151
 
550
- | Source | Edge Types | Trigger | Notes |
551
- |--------|-----------|---------|-------|
552
- | A-MEM `generateMemoryLinks()` | semantic, supporting, contradicts | Indexing (new docs only) | LLM-assessed confidence + reasoning. Requires embeddings for neighbor discovery. |
553
- | A-MEM `inferCausalLinks()` | causal | Post-response (IO3 decision-extractor) | Links between `_clawmem/agent/observations/` docs, not arbitrary workspace docs. |
554
- | Beads `syncBeadsIssues()` | causal, supporting, semantic | `beads_sync` MCP tool or watcher (.beads/ change) | Queries `bd` CLI (Dolt backend). Maps beads deps: blocks→causal, discovered-from→supporting, relates-to→semantic, plus conditional-blocks→causal, caused-by→causal, supersedes→supporting. Metadata: `{origin: "beads"}`. |
555
- | `buildTemporalBackbone()` | temporal | `build_graphs` MCP tool (manual) | Creation-order edges between all active docs. |
556
- | `buildSemanticGraph()` | semantic | `build_graphs` MCP tool (manual) | Pure cosine similarity. PK collision: `INSERT OR IGNORE` means A-MEM semantic edges take precedence if they exist first. |
557
- | Entity co-occurrence graph | entity | A-MEM enrichment (indexing) | LLM entity extraction → quality filters (title/length/blocklist/location validation) → type-agnostic canonical resolution within compatibility buckets (person, org, location, tech=project/service/tool/concept) → `entity_mentions` + `entity_cooccurrences` tables. Entity edges use IDF-based specificity scoring. Feeds ENTITY intent queries and MPFP `[entity, semantic]` patterns. |
558
- | `consolidated_observations` | supporting, contradicts | Consolidation worker (background, light + heavy lanes) | 3-tier consolidation: facts → observations → mental models. Observations track `proof_count`, `trend` (STABLE/STRENGTHENING/WEAKENING/STALE), and source links. **v0.7.1 safety gates:** name-aware merge gate uses entity-anchor comparison + 3-gram cosine similarity (dual-threshold `CLAWMEM_MERGE_SCORE_NORMAL`=0.93 / `_STRICT`=0.98) to prevent cross-entity merges ("Alice decided X" merging into "Bob decided X"). Merge-time contradiction gate runs deterministic heuristic + LLM check; blocked merges route to `CLAWMEM_CONTRADICTION_POLICY`=`link` (new row + `contradicts` edge, default) or `supersede` (old row `status='inactive'`, new row replaces). **v0.8.0:** the heavy lane calls `consolidateObservations(store, llm, { maxDocs, guarded: true, staleOnly: true })` — `guarded: true` forces merge-safety enforcement regardless of `CLAWMEM_MERGE_GUARD_DRY_RUN`, and `staleOnly: true` reorders candidates by `recall_stats.last_recalled_at ASC` so long-unseen docs bubble up first. Optional `candidateIds` filter plumbs k-NN anomaly ids from `computeSurprisalScores`. |
559
- | Deductive synthesis | supporting, contradicts | Consolidation worker Phase 3 (background, every ~15 min in the light lane; batched in the heavy lane) | Combines 2-3 related recent observations (decision/preference/milestone/problem, last 7 days) into `content_type='deductive'` documents with `source_doc_ids` provenance. First-class searchable docs with ∞ half-life. **v0.7.1 anti-contamination wrapper:** every draft passes through deterministic pre-checks (empty conclusion, invalid source_indices, pool-only entity contamination via `entity_mentions` or lexical fallback) + LLM validator (fail-open with `validatorFallbackAccepts` counter) + dedupe. Per-reason rejection stats exposed via `DeductiveSynthesisStats` (contaminationRejects, invalidIndexRejects, unsupportedRejects, emptyRejects, dedupSkipped, validatorFallbackAccepts). Contradictory dedupe matches are linked via `contradicts` edges. **v0.8.0:** heavy lane calls `generateDeductiveObservations(store, llm, { maxRecent, guarded: true, staleOnly: true })` for stale-first batching on large vaults. |
560
- | Heavy maintenance lane journal | — | `clawmem` process with `CLAWMEM_HEAVY_LANE=true` (v0.8.0) | Writes a row to the `maintenance_runs` table for every scheduled heavy-lane attempt (including skips). Columns: `lane` (`heavy`), `phase` (`gate`/`consolidate`/`deductive`), `status` (`started`/`completed`/`failed`/`skipped`), `reason` (`outside_window`/`query_rate_high`/`lease_unavailable`), per-phase `selected`/`processed`/`created`/`null_call` counts, and `metrics_json` with selector type (`stale-first`/`surprisal`/`surprisal-fallback-stale`) + full `DeductiveSynthesisStats` on completed Phase 3 runs. Exclusivity is enforced via the new `worker_leases` table with atomic `INSERT ... ON CONFLICT DO UPDATE ... WHERE expires_at <= ?` acquisition, random 16-byte fencing tokens, and TTL reclaim. Gate uses the existing `context_usage` table (no `query_activity` table) and `recall_stats.last_recalled_at` for stale-first ordering. Off by default — requires explicit opt-in. |
561
- | Conversation synthesis (`runConversationSynthesis`) | semantic, supporting, contradicts, causal, temporal, entity | `clawmem mine <dir> --synthesize` (opt-in, post-index) | **v0.7.2.** Two-pass LLM pipeline over freshly imported `content_type='conversation'` docs. Pass 1 extracts structured facts (decision/preference/milestone/problem) via `extractFactsFromConversation`, saves each via dedup-aware `saveMemory`, populates a local Set-based alias map. Pass 2 resolves cross-fact links against the local map first (fails closed on ambiguity — multi-candidate titles return unresolved), falls back to collection-scoped SQL lookup with LIMIT 2 ambiguity detection. Relations upsert via `ON CONFLICT DO UPDATE SET weight = MAX(weight, excluded.weight)` — idempotent on equal-weight reruns but monotonically accepts stronger later evidence. Synthesized fact paths are a pure function of `(sourceDocId, slug(title), short sha256(normalizedTitle))`, so reruns update in place instead of creating parallel rows. Counters split into `llmFailures` (null/thrown/invalid-JSON) vs `docsWithNoFacts` (valid-empty extraction). All failures non-fatal — never rolls back the mine import. |
152
+ ## Indexing rules
562
153
 
563
- **Edge collision:** Both `generateMemoryLinks()` and `buildSemanticGraph()` insert `relation_type='semantic'`. PK is `(source_id, target_id, relation_type)` first writer wins.
154
+ - **Indexed** (per collection in `config.yaml`): `**/MEMORY.md` · `**/memory/**` · `**/docs/**` · `**/research/**` · `**/YYYY-MM-DD*` (`.md`/`.txt`).
155
+ - **Excluded (always):** `gits/`, `scraped/`, `.git/`, `node_modules/`, `dist/`, `build/`, `vendor/`. **Never index** credential files (`.env`, `*secrets*`, `*credentials*`) or `gits/`.
156
+ - **Indexing ≠ embedding:** the watcher indexes on `.md` change but does NOT embed; the embed timer (or `clawmem embed`) keeps vectors fresh. Missing embeddings silently degrade vector recall — BM25 still works.
564
157
 
565
- **Graph traversal asymmetry:** `adaptiveTraversal()` traverses all edge types outbound (source→target) but only `semantic` and `entity` edges inbound (target→source). Temporal and causal edges are directional only.
158
+ architecture + graph building: [docs/concepts/architecture.md](docs/concepts/architecture.md), [docs/internals/graph-traversal.md](docs/internals/graph-traversal.md).
566
159
 
567
- **MPFP graph retrieval (v0.2.0):** Multi-Path Fact Propagation runs predefined meta-path patterns in parallel (`[semantic, semantic]`, `[entity, temporal]`, `[semantic, causal]`, etc.), fuses via **max-score** (best supporting path wins). Note: meta-path fusion is max-score, NOT RRF — Forward Push yields absolute propagation mass where magnitude carries signal, so rank-only fusion would discard the difference between a strong path hit and a barely-surviving tail hit (`src/graph-traversal.ts:413-425`). This is distinct from the `query`/`intent_search` *outer* retrieval, which does fuse BM25+vector via RRF. Hop-synchronized edge cache batches DB queries per hop instead of per pattern. Forward Push with α=0.15 teleport probability bounds active nodes sublinearly. Tier 3 only (`query`/`intent_search`), not hooks. Patterns selected per MAGMA intent: WHY → `[semantic, causal]`, ENTITY → `[entity, semantic]`, WHEN → `[temporal, semantic]`.
160
+ ---
568
161
 
569
- ### When to Run `build_graphs`
162
+ ## Memory lifecycle (pin / snooze / forget — manual tools)
570
163
 
571
- - After **bulk ingestion** (many new docs at once) — adds temporal backbone and fills semantic gaps where A-MEM links are sparse.
572
- - When `intent_search` for WHY/ENTITY returns **weak or obviously incomplete results** and you suspect graph sparsity.
573
- - Do NOT run after every reindex. Routine indexing creates A-MEM links automatically for new docs.
164
+ - **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
165
+ - **`memory_snooze`** PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
166
+ - **`memory_forget`** only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
167
+ - **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.
574
168
 
575
- ### When to Run `index_stats`
169
+ ---
576
170
 
577
- - After bulk ingestion to verify doc counts and embedding coverage.
578
- - When retrieval quality seems degraded — check for unembedded docs or content type distribution issues.
579
- - Do NOT run routinely.
171
+ ## Operational gotchas
580
172
 
581
- ## Pipeline Details
173
+ - **Empty `context-surfacing`** → prompt < 20 chars, starts with `/`, or nothing scored above threshold. Check `clawmem status` + embedding coverage.
174
+ - **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed`.
175
+ - **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs`. Don't run it after every reindex (A-MEM links per-doc automatically).
176
+ - **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert scores, silently collapsing ranking to RRF.
177
+ - **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → almost always the context-surfacing hook's **cold-start** (fresh Bun process + opening a large `index.sqlite` + cold OS page cache), NOT inference. Recurs on memory-constrained hosts (e.g. WSL with a low memory cap) as the cache is evicted between turns. **Durable fix: give the host enough RAM to keep the index cached**; raising the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) is a secondary margin — avoid 15s+ as a standing default. Detail: [docs/troubleshooting.md](docs/troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
178
+ - **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [docs/troubleshooting.md](docs/troubleshooting.md).
582
179
 
583
- ### `query` (default Tier 3 workhorse)
180
+ ---
584
181
 
585
- ```
586
- User Query + optional intent hint
587
- → Temporal Extraction (regex date range from query: "last week", "March 2026" → WHERE modified_at BETWEEN filters)
588
- → BM25 Probe → Strong Signal Check (skip expansion if top hit ≥ 0.85 with gap ≥ 0.15; disabled when intent provided)
589
- → Query Expansion (LLM generates text variants; intent steers expansion prompt)
590
- → Parallel (typed routing): BM25(original) + Vector(original) + BM25(lex expansions) + Vector(vec/hyde expansions)
591
- + Temporal Proximity (date-range filtered, if temporal constraint extracted)
592
- + Entity Graph (conditional 1-hop entity walk from top seeds, if entity signals present)
593
- → Original query lists get positional 2× weight in RRF; expanded get 1×
594
- → Reciprocal Rank Fusion (k=60, top candidateLimit, up to 4-way parallel legs)
595
- → Intent-Aware Chunk Selection (intent terms at 0.5× weight alongside query terms at 1.0×)
596
- → Cross-Encoder Reranking (4000 char context; intent prepended to rerank query; chunk dedup; batch cap=4)
597
- → Rerank/RRF Blend (blendRerank: 0.9·reranker + 0.1·normalized-RRF tiebreaker; reranker can promote over RRF #1; falls back to RRF order if reranker unavailable)
598
- → SAME Composite Scoring
599
- → MMR Diversity Filter (Jaccard bigram similarity > 0.6 → demoted, not removed)
600
- ```
182
+ ## Anti-patterns
601
183
 
602
- ### `intent_search` (specialist for causal chains)
184
+ - ❌ Manually pick `query`/`intent_search`/`search` when `memory_retrieve` can auto-route → ✅ `memory_retrieve` first.
185
+ - ❌ Call MCP tools every turn → ✅ only when the 3-rule gate fires.
186
+ - ❌ Re-search what's already in `<vault-context>`.
187
+ - ❌ Run `status` routinely → ✅ only when retrieval feels broken or after large ingestion.
188
+ - ❌ Pin everything → ✅ pin only persistent high-priority items.
189
+ - ❌ Forget memories to "clean up" → ✅ let decay + contradiction detection handle it.
190
+ - ❌ `build_graphs` after every reindex → ✅ only after bulk ingestion or weak graph traversal.
191
+ - ❌ `diary_write` in Claude Code → ✅ hooks capture this (diary is for non-hooked envs only).
192
+ - ❌ `kg_query` for causal "why" → ✅ `intent_search` (kg_query is entity facts, not reasoning chains).
603
193
 
604
- ```
605
- User Query → Intent Classification (WHY/WHEN/ENTITY/WHAT)
606
- → BM25 + Vector (intent-weighted RRF: boost BM25 for WHEN, vector for WHY)
607
- → Graph Traversal (WHY/ENTITY only; multi-hop beam search over memory_relations)
608
- Outbound: all edge types (semantic, supporting, contradicts, causal, temporal)
609
- Inbound: semantic and entity only
610
- Scores normalized to [0,1] before merge with search results
611
- → Cross-Encoder Reranking (200 char context per doc; file-keyed score join)
612
- → SAME Composite Scoring (uses stored confidence from contradiction detection + feedback)
613
- ```
194
+ ---
614
195
 
615
- ### Key Differences
196
+ ## Curator agent
616
197
 
617
- | Aspect | `query` | `intent_search` |
618
- |--------|---------|-----------------|
619
- | Query expansion | Yes (skipped on strong BM25 signal) | No |
620
- | Intent hint | Yes (`intent` param steers 5 stages) | Auto-detected (WHY/WHEN/ENTITY/WHAT) |
621
- | Rerank context | 4000 chars/doc (intent-aware chunk selection) | 200 chars/doc |
622
- | Chunk dedup | Yes (identical texts share single rerank call) | No |
623
- | Graph traversal | No | Yes (WHY/ENTITY, multi-hop) |
624
- | MMR diversity | Yes (`diverse=true` default) | No |
625
- | `compact` param | Yes | No |
626
- | `collection` filter | Yes | No |
627
- | `candidateLimit` | Yes (default 30) | No |
628
- | Best for | Most queries, progressive disclosure | Causal chains spanning multiple docs |
198
+ Maintenance agent for Tier-3 work the main agent neglects. Install: `clawmem setup curator`. Invoke: **"curate memory" / "run curator" / "memory maintenance"**. Six phases: health snapshot → lifecycle triage (pin/snooze/propose-forget, never auto-confirms) → retrieval health probes → reflect + consolidate `--dry-run` → conditional graph rebuild → collection hygiene. Never auto-confirms forget, never runs embed, never edits config.
629
199
 
200
+ ---
630
201
 
631
- ## Troubleshooting
202
+ ## Integrations
632
203
 
633
- ```
634
- Symptom: "Local model download blocked" error
635
- → llama-server endpoint unreachable while CLAWMEM_NO_LOCAL_MODELS=true.
636
- Fix: Start the llama-server instance. Or set CLAWMEM_NO_LOCAL_MODELS=false for in-process fallback.
637
-
638
- Symptom: "[generate] Remote LLM in cooldown, falling back to in-process generation"
639
- → Remote LLM server had a transport failure (ECONNREFUSED/ETIMEDOUT). ClawMem set a 60s cooldown
640
- and is using local node-llama-cpp. Remote will be retried after cooldown expires.
641
- → Not an error if you expect local fallback. If you want remote only: ensure llama-server is running,
642
- or set CLAWMEM_NO_LOCAL_MODELS=true to get null instead of slow local inference.
643
-
644
- Symptom: Query expansion always fails / returns garbage
645
- → On CPU-only systems, in-process inference is significantly slower and less reliable. Systems with GPU acceleration (Metal/Vulkan) handle these models well in-process.
646
- → Fix: Run llama-server on a GPU. Even a low-end NVIDIA card handles 1.7B models.
647
-
648
- Symptom: Vector search returns no results but BM25 works
649
- → Missing embeddings. Watcher indexes but does NOT embed.
650
- → Fix: Run `clawmem embed` or wait for the daily embed timer.
651
-
652
- Symptom: llama-server crashes with "non-causal attention requires n_ubatch >= n_tokens"
653
- → Embedding/reranking models use non-causal attention. When -b (batch) > -ub (ubatch), the assertion fails.
654
- → Fix: Set -ub equal to -b (e.g. -b 2048 -ub 2048). Never omit -ub for embedding/reranking servers.
655
-
656
- Symptom: context-surfacing hook returns empty
657
- → Prompt too short (<20 chars), starts with `/`, or no docs score above threshold.
658
- → Fix: Check `clawmem status` for doc counts. Check `clawmem embed` for embedding coverage.
659
-
660
- Symptom: intent_search returns weak results for WHY/ENTITY
661
- → Graph may be sparse (few A-MEM edges).
662
- → Fix: Run `build_graphs` to add temporal backbone + semantic edges.
663
-
664
- Symptom: Watcher logs events but collections show 0 docs after update/reindex
665
- → Bun.Glob does not support brace expansion {a,b,c}. Collection patterns returned 0 files.
666
- → Fixed 2026-02-12: indexer.ts splits brace patterns into individual Glob scans.
667
-
668
- Symptom: Watcher fires events but wrong collection processes them (e.g., workspace instead of dharma-propagation)
669
- → Collection prefix matching via Array.find() returns first match. Parent paths match before children.
670
- → Fixed 2026-02-12: cmdWatch() sorts collections by path length descending (most specific first).
671
-
672
- Symptom: reindex --force crashes with "UNIQUE constraint failed: documents.collection, documents.path"
673
- → Force deactivates rows (active=0) but UNIQUE(collection, path) doesn't discriminate by active flag.
674
- → Fixed 2026-02-12: indexer.ts checks for inactive rows and reactivates instead of inserting.
675
-
676
- Symptom: embed crashes with "UNIQUE constraint failed on vectors_vec primary key" on restart
677
- → vectors_vec is a vec0 virtual table — INSERT OR REPLACE is not supported by vec0.
678
- → Fixed 2026-03-15: insertEmbedding() uses DELETE (try-catch) + INSERT instead of INSERT OR REPLACE.
679
- → Embed can now resume after interrupted runs without --force.
680
-
681
- Symptom: embed crashes with alternating "no such table: vectors_vec" / "table vectors_vec already exists"
682
- → Dimension migration race: --force drops vectors_vec, ensureVecTable per-fragment drops+recreates on dimension
683
- mismatch, causing rapid table existence flickering between fragments.
684
- → Fixed 2026-03-15: ensureVecTable caches verified dimensions (vecTableDims), uses CREATE VIRTUAL TABLE IF NOT EXISTS,
685
- and clearAllEmbeddings resets the cache. First fragment creates, rest skip the check.
686
-
687
- Symptom: embed --force with new model produces 3 docs stuck as "Unembedded" but "All documents already embedded"
688
- → First fragment (seq=0) failed during a crashed embed run. Later fragments succeeded.
689
- getHashesNeedingFragments thinks the doc is done but status checks seq=0 specifically.
690
- → Fix: Delete partial content_vectors + vectors_vec for the stuck hashes, then re-run embed (no --force).
691
- The vec0 DELETE try-catch prevents cascading failures during the re-embed.
692
-
693
- Symptom: reindex --force after v0.2.0 upgrade shows no entity extraction
694
- → `reindex --force` treats existing docs as updates (isNew=false). The A-MEM pipeline
695
- skips entity extraction, link generation, and evolution for updates to avoid churn.
696
- → Fix: Use `clawmem reindex --enrich` instead. The `--enrich` flag forces the full
697
- enrichment pipeline (entity extraction + links + evolution) on all documents.
698
- → `--force` alone only refreshes A-MEM notes (keywords, tags, context). `--enrich`
699
- is needed after major upgrades that add new enrichment stages.
700
-
701
- Symptom: `clawmem update` crashes with "Binding expected string, TypedArray, boolean, number, bigint or null"
702
- → YAML frontmatter values like `title: 2023-09-27` or `title: true` are coerced by gray-matter
703
- into Date objects or booleans. Bun's SQLite driver rejects these as bind parameters.
704
- → Fixed v0.4.2: `parseDocument()` runtime-checks all frontmatter fields via `str()` helper.
705
- Defense-in-depth `safeTitle` guards in `insertDocument`/`updateDocument`/`reactivateDocument`.
706
- → Affects: title, domain, workstream, content_type, review_by — any field gray-matter can coerce.
707
-
708
- Symptom: CLI reindex/update falls back to node-llama-cpp Vulkan (not GPU server)
709
- → GPU env vars only in systemd drop-in, not in wrapper script. CLI invocations missed them.
710
- → Fixed 2026-02-12: bin/clawmem wrapper exports CLAWMEM_EMBED_URL/LLM_URL/RERANK_URL defaults.
711
- → Always run ClawMem via the `bin/clawmem` wrapper, not `bun run src/clawmem.ts` directly.
712
- The wrapper sets CLAWMEM_EMBED_URL/LLM_URL/RERANK_URL defaults. Scripts or inline bun
713
- commands that bypass the wrapper will fall back to in-process node-llama-cpp (slow, CPU).
714
-
715
- Symptom: "UserPromptSubmit hook error" on context-surfacing hook (intermittent)
716
- → SQLite contention between the watcher and the hook. The watcher processes filesystem events
717
- and holds brief write locks. If the hook fires during a lock, it can exceed its timeout.
718
- More likely during active conversations with frequent file changes.
719
- → v0.1.6 fix: watcher no longer processes session transcript .jsonl files (only .beads/*.jsonl),
720
- eliminating the most common source of contention.
721
- → Default hook timeout is 8s (since v0.1.1). If you have an older install, re-run
722
- `clawmem setup hooks`. If persistent, restart the watcher: `systemctl --user restart
723
- clawmem-watcher.service`. Healthy memory is under 100MB — if 400MB+, restart clears it.
724
- → v0.2.4 fix: hook's SQLite busy_timeout was 500ms — too tight. During A-MEM enrichment
725
- or heavy indexing, watcher write locks exceed 500ms, causing SQLITE_BUSY. Raised to
726
- 5000ms (matches MCP server). Still completes within the 8s outer timeout.
727
-
728
- Symptom: WSL hangs or becomes unresponsive during long sessions / watcher has 100K+ FDs
729
- → Pre-v0.2.3: fs.watch(recursive: true) registered inotify watches on EVERY subdirectory,
730
- including excluded dirs (gits/, node_modules/, .git/). Broad collection paths like
731
- ~/Projects with 67K subdirs exhausted inotify limits.
732
- → v0.2.3 fix: watcher walks dir trees at startup, skips excluded subtrees, watches
733
- non-excluded dirs individually. 500-dir cap per collection path.
734
- → Diagnosis: `ls /proc/$(pgrep -f "clawmem.*watch")/fd | wc -l` — healthy < 15K.
735
- → If still high: narrow broad collection paths. See docs/troubleshooting.md for details.
736
- ```
204
+ - **Claude Code** — `clawmem setup hooks && clawmem setup mcp`. Hooks = 90% auto; 31 MCP tools = 10%.
205
+ - **OpenClaw** — native memory plugin (`kind: memory`, v0.10.0+): `clawmem setup openclaw`. → [docs/guides/openclaw-plugin.md](docs/guides/openclaw-plugin.md).
206
+ - **Hermes** `MemoryProvider` plugin: copy `src/hermes/` into `$HERMES_HOME/plugins/clawmem/`. → [docs/guides/hermes-plugin.md](docs/guides/hermes-plugin.md).
207
+ - **REST API** `clawmem serve [--port 7438]`. [docs/reference/rest-api.md](docs/reference/rest-api.md).
737
208
 
738
- ## CLI Reference
209
+ All integrations share the same SQLite vault — decisions captured in one runtime surface in the others.
739
210
 
740
- Run `clawmem --help` for full command listing. Use this before guessing at commands or parameters.
211
+ ---
741
212
 
742
- **IO6 surface commands** (for daemon/`--print` mode integration):
743
- ```bash
744
- # IO6a: per-prompt context injection (pipe prompt on stdin)
745
- echo "user query" | clawmem surface --context --stdin
213
+ ## Tool selection (one-liner)
746
214
 
747
- # IO6b: per-session bootstrap injection (pipe session ID on stdin)
748
- echo "session-id" | clawmem surface --bootstrap --stdin
749
215
  ```
750
-
751
- **Browse and analysis:**
752
- ```bash
753
- clawmem list [-n N] [-c col] # Browse recent documents (--json for machine output)
754
- clawmem reflect [N] # Cross-session reflection (last N days, default 14)
755
- # Shows recurring themes, antipatterns, co-activation clusters
756
- clawmem consolidate [--dry-run] # Find and archive duplicate low-confidence documents
757
- # Uses Jaccard similarity within same collection
216
+ memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) -> multi_get -> search/vsearch (spot checks)
758
217
  ```
759
218
 
760
- **Session focus topic (v0.9.0 §11.4):** Per-session topic biasing for context-surfacing. Writes a focus file at `~/.cache/clawmem/sessions/<session_id>.focus` that steers query expansion, reranking, snippet extraction, and post-composite-score topic boost (1.4× match / 0.75× demote, NO-OP on zero matches). Session-isolated — never writes to SQLite or lifecycle columns. The session ID is read from `--session-id <id>`, then `CLAUDE_SESSION_ID`, then `CLAWMEM_SESSION_ID`. When to use: user says "focus on authentication for this session" / "only surface X-related docs right now" / "let's work on Y this session." Clear the focus at the end of the subsession to return to baseline surfacing.
761
-
762
- ```bash
763
- # Set a focus topic for the current session (multi-word OK)
764
- clawmem focus set "authentication flow" # uses CLAUDE_SESSION_ID / CLAWMEM_SESSION_ID env var
765
- clawmem focus set "authentication flow" --session-id abc123 # explicit
766
-
767
- # Show / clear
768
- clawmem focus show --session-id abc123
769
- clawmem focus clear --session-id abc123
770
- ```
219
+ ---
771
220
 
772
- ## Integration Notes
773
-
774
- - **Memory nudge (v0.2.0):** Every N prompts (default 15) without a lifecycle MCP tool call (`memory_pin`/`memory_forget`/`memory_snooze`), context-surfacing appends `<vault-nudge>` prompting proactive memory management. Counter resets on lifecycle tool use. Configure via `CLAWMEM_NUDGE_INTERVAL` (0 to disable).
775
- - **Entity resolution (v0.2.0+):** A-MEM enrichment extracts named entities via LLM, resolves to canonical forms using FTS5 + Levenshtein fuzzy matching with **type-agnostic compatibility buckets** (person, org, location stay separate; project/service/tool/concept merge freely as "tech" bucket). Quality filters reject title-as-entity, long names, template placeholders, and invalid locations. Entity edges use IDF-based specificity scoring (rare entities create edges; ubiquitous entities alone cannot). See `docs/internals/entity-resolution.md` for customization (extending type vocabulary and buckets).
776
- - QMD retrieval (BM25, vector, RRF, rerank, query expansion) is forked into ClawMem. Do not call standalone QMD tools.
777
- - SAME (composite scoring), MAGMA (intent + graph), A-MEM (self-evolving notes) layer on top of QMD substrate.
778
- - Three inference services (embedding, LLM, reranker) on local or remote GPU — `llama-server` for all three in the default stack; the SOTA stack serves the reranker as a transformers sidecar. Wrapper defaults to `localhost:8088/8089/8090`.
779
- - `CLAWMEM_NO_LOCAL_MODELS=false` (default) allows in-process LLM/reranker fallback via `node-llama-cpp`. Set `true` for remote-only setups to fail fast on unreachable endpoints.
780
- - Consolidation worker (`CLAWMEM_ENABLE_CONSOLIDATION=true`) backfills unenriched docs with A-MEM notes + links. Only runs if the MCP process stays alive long enough to tick (every 5min).
781
- - Beads integration: `syncBeadsIssues()` queries `bd` CLI (Dolt backend, v0.58.0+) for live issue data, creates markdown docs in `beads` collection, maps all dependency edge types into `memory_relations`, and triggers A-MEM enrichment for new docs. Watcher auto-triggers on `.beads/` directory changes; `beads_sync` MCP tool for manual sync. Requires `bd` binary on PATH or at `~/go/bin/bd`.
782
- - HTTP REST API: `clawmem serve [--port 7438]` — optional REST server on localhost. Search, retrieval, lifecycle, and graph traversal. `POST /retrieve` mirrors `memory_retrieve` with auto-routing (keyword/semantic/causal/timeline/hybrid). `POST /search` provides direct mode selection. Bearer token auth via `CLAWMEM_API_TOKEN` env var (disabled if unset).
783
- - OpenClaw memory plugin: `clawmem setup openclaw` — registers ClawMem as a native OpenClaw memory plugin (`kind: memory`, v0.10.0+). Lifecycle events on the plugin-hook bus: `before_prompt_build` is the **load-bearing** path — it runs prompt-aware retrieval (context-surfacing) AND the pre-emptive `precompact-extract` synchronously when token usage approaches the compaction threshold, so state is captured BEFORE the LLM call that could trigger compaction; `agent_end` runs decision-extractor + handoff-generator + feedback-loop in parallel (fire-and-forget at OpenClaw, plus a 30s default void-hook timeout from OpenClaw v2026.4.26+ that logs slow handlers but does not cancel the underlying postrun work); `before_compaction` is **defense-in-depth fallback only** — fire-and-forget, races the compactor, exists for the rare case where the `before_prompt_build` proximity heuristic missed a sudden token jump; `session_start` registers the session and caches first-turn bootstrap context. Shares the same vault as Claude Code hooks (dual-mode). SQLite busy_timeout=5000ms for concurrent access safety.
784
- - **§14.3 pure-memory migration (v0.10.0):** v0.10.0 drops the `ClawMemContextEngine` class entirely. Previous versions registered as `kind: context-engine` and implemented `assemble()`/`bootstrap()`/`afterTurn()`/`compact()` on a class. v0.10.0 registers as `kind: memory` and wires every lifecycle surface through plugin hooks on the event bus. Retrieval pipeline, composite scoring, vault format, and the 5 registered agent tools are unchanged — this is a packaging and registration change, not a behavioral one.
785
- - **v2026.4.11 packaging fix (v0.10.0):** `src/openclaw/package.json` declares `openclaw.extensions: ["./index.ts"]` (required by v2026.4.11's discovery path), and `cmdSetupOpenClaw` defaults to `cpSync(..., { recursive: true, dereference: true })` because v2026.4.11's discoverer uses `readdirSync({ withFileTypes: true })` where symlink `isDirectory() === false`. A `--link` opt-in flag preserves the old symlink behavior for dev workflows with a warning.
786
- - **Profile-aware install (v0.10.4, §28.1, issue #11):** `cmdSetupOpenClaw` is now a three-path installer. When `openclaw` is on `PATH` it delegates to `openclaw plugins install <pluginDir> --force` (or `-l` for `--link`), inheriting OpenClaw's destination resolution (`OPENCLAW_STATE_DIR`, `OPENCLAW_CONFIG_PATH`, `--profile`), manifest validation, security scan, install records, slot selection, and registry refresh. The plugin is auto-enabled — Path 1's "Next steps" output drops `openclaw plugins enable clawmem` because it's redundant. `--link` in delegated mode records the source in `plugins.load.paths` (NOT a filesystem symlink, so the v2026.4.11 discovery skip does NOT apply). When `openclaw` is absent, ClawMem falls back to recursive copy at a destination resolved by `src/openclaw-paths.ts:resolveExtensionsDirNoOpenClaw` — a faithful mirror of OpenClaw's `resolveConfigDir` (priority: `OPENCLAW_STATE_DIR` → `OPENCLAW_CONFIG_PATH` → `OPENCLAW_HOME`/`HOME`/`USERPROFILE`/`os.homedir()`/`cwd` → `~/.openclaw`). The fallback's filesystem symlink in `--link` mode is still subject to OpenClaw's discovery skip (warning surfaced). `--remove` tries `openclaw plugins uninstall clawmem --force` first; on failure (typical for legacy unmanaged direct-copy installs), warns the user that config/install records may need manual repair, then runs manual cleanup at the resolved path. On success, also performs a constrained stale cleanup of any leftover unmanaged directory at the same path. `--help` / `-h` short-circuits before any spawn or filesystem work and prints the full flag + env-var reference (§28.2). Helpers in `src/openclaw-paths.ts` take injected `env` + `homedir` for testability.
787
- - **v2026.4.18 synchronous-`register()` constraint:** OpenClaw v2026.4.18 (`fix(plugins): enforce synchronous registration`) throws `"plugin register must be synchronous"` if the plugin's `register()` function returns a Promise. ClawMem's `register(api)` in `src/openclaw/index.ts` is intentionally synchronous — all `await` work lives inside per-event handlers, never in registration itself. Companion change: register failures now atomically roll back side effects (globals, hook registrations, tool registrations), so any future throw inside `register()` will leave OpenClaw in a clean state. Keep the function synchronous and throw-free; do not add `async` or top-level `await`.
788
- - **Precompact correctness contract (v0.10.0):** The load-bearing precompact path is `before_prompt_build`, NOT `before_compaction`. `before_prompt_build` is awaited synchronously before the LLM call that could trigger compaction, so it cannot race the compactor. `before_compaction` is fire-and-forget at OpenClaw's call site and exists only as a safety net for the rare case the proximity heuristic in `before_prompt_build` missed a sudden token-count jump. Do not describe `before_compaction` as the primary surface — the guarantee comes from `before_prompt_build`. v0.3.0 did the pre-emptive extraction from `ContextEngine.compact()` via `delegateCompactionToRuntime()`; v0.10.0 moves it into `before_prompt_build` where it can be awaited before the triggering LLM call. User-visible behavior is equivalent or better: state capture now happens strictly before compaction, not in a race with it.
789
- - Hermes Agent MemoryProvider plugin: `src/hermes/` — Python plugin implementing Hermes's `MemoryProvider` ABC. **Preferred install:** copy into `$HERMES_HOME/plugins/clawmem/` (typically `~/.hermes/plugins/clawmem/`) Hermes #10529 (v2026.4.13+) added user-plugin discovery, so this path survives `git pull` of hermes-agent. **Bundled-style install:** `hermes-agent/plugins/memory/clawmem/` still works (bundled-first precedence on name collisions). Uses shell-out for lifecycle hooks (session-bootstrap, context-surfacing, extraction) and REST API for tools (retrieve, get, session_log, timeline, similar). Plugin manages its own transcript JSONL for ClawMem hooks. Supports external (you run `clawmem serve`) and managed (plugin starts/stops serve) modes. **Agent-context isolation:** `initialize()` reads the `agent_context` kwarg Hermes passes ("primary"/"subagent"/"cron"/"flush"); for non-primary contexts the read-side hooks (session-bootstrap, context-surfacing) still run but the write-side surfaces (`sync_turn` transcript appends, `on_session_end` extraction, `on_pre_compress` precompact) early-return so cron system prompts and subagent intermediate state never reach the vault.
221
+ ## Deep reference (docs/)
222
+
223
+ | Topic | Path |
224
+ |---|---|
225
+ | Quickstart / introduction | `docs/quickstart.md`, `docs/introduction.md` |
226
+ | Inference stack + server setup | `docs/guides/inference-services.md` |
227
+ | Cloud embedding | `docs/guides/cloud-embedding.md` |
228
+ | Configuration (env vars) | `docs/reference/configuration.md` |
229
+ | Hooks vs MCP | `docs/concepts/hooks-vs-mcp.md` |
230
+ | Composite scoring | `docs/concepts/composite-scoring.md` |
231
+ | Architecture / multi-vault | `docs/concepts/architecture.md`, `docs/concepts/multi-vault.md` |
232
+ | Pipelines / graph / entities | `docs/internals/{query-pipeline,intent-search-pipeline,graph-traversal,entity-resolution}.md` |
233
+ | MCP tools / CLI / REST | `docs/reference/{mcp-tools,cli,rest-api}.md` |
234
+ | Setup (hooks / mcp / systemd) | `docs/guides/{setup-hooks,setup-mcp,systemd-services}.md` |
235
+ | OpenClaw / Hermes plugins | `docs/guides/openclaw-plugin.md`, `docs/guides/hermes-plugin.md` |
236
+ | Troubleshooting | `docs/troubleshooting.md` |
237
+ | Upgrading | `docs/guides/upgrading.md` |
238
+ | Operating ClawMem at query time (portable skill) | `SKILL.md` |