clawmem 0.13.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +165 -677
- package/CLAUDE.md +4 -747
- package/README.md +20 -191
- package/SKILL.md +157 -711
- package/docs/clawmem-architecture.excalidraw +2415 -0
- package/docs/clawmem-architecture.png +0 -0
- package/docs/clawmem_hero.jpg +0 -0
- package/docs/concepts/architecture.md +413 -0
- package/docs/concepts/composite-scoring.md +133 -0
- package/docs/concepts/hooks-vs-mcp.md +156 -0
- package/docs/concepts/multi-vault.md +71 -0
- package/docs/contributing.md +101 -0
- package/docs/guides/cloud-embedding.md +134 -0
- package/docs/guides/hermes-plugin.md +187 -0
- package/docs/guides/inference-services.md +144 -0
- package/docs/guides/multi-vault-config.md +84 -0
- package/docs/guides/openclaw-plugin.md +306 -0
- package/docs/guides/setup-hooks.md +146 -0
- package/docs/guides/setup-mcp.md +76 -0
- package/docs/guides/systemd-services.md +332 -0
- package/docs/guides/upgrading.md +566 -0
- package/docs/internals/entity-resolution.md +135 -0
- package/docs/internals/graph-traversal.md +85 -0
- package/docs/internals/intent-search-pipeline.md +103 -0
- package/docs/internals/query-pipeline.md +100 -0
- package/docs/introduction.md +104 -0
- package/docs/quickstart.md +158 -0
- package/docs/reference/cli.md +195 -0
- package/docs/reference/configuration.md +101 -0
- package/docs/reference/mcp-tools.md +336 -0
- package/docs/reference/rest-api.md +204 -0
- package/docs/troubleshooting.md +330 -0
- package/package.json +2 -1
- package/src/clawmem.ts +60 -0
- package/src/health/rerank-golden.json +54 -0
- package/src/health/rerank-health.ts +150 -0
- package/src/mcp.ts +20 -1
- package/src/memory.ts +2 -0
- package/src/search-utils.ts +35 -4
- package/src/store.ts +115 -22
package/AGENTS.md
CHANGED
|
@@ -1,750 +1,238 @@
|
|
|
1
1
|
# ClawMem — Agent Quick Reference
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
On-device retrieval-augmented memory for Claude Code, OpenClaw, and Hermes agents. **Hooks** auto-inject context (~90%); **MCP tools** cover targeted recall (~10%). TypeScript on Bun, MIT.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
This file is the **lean root SSOT** — agent-facing essentials only. Deep reference lives in `docs/` (linked throughout, indexed at the bottom). `CLAUDE.md` is an `@AGENTS.md` import; `SKILL.md` is the portable on-demand operating reference.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
| Service | Port | Model | VRAM | Protocol |
|
|
10
|
-
|---|---|---|---|---|
|
|
11
|
-
| Embedding | 8088 | EmbeddingGemma-300M-Q8_0 | ~400MB | `/v1/embeddings` |
|
|
12
|
-
| LLM | 8089 | qmd-query-expansion-1.7B-q4_k_m | ~2.2GB | `/v1/chat/completions` |
|
|
13
|
-
| Reranker | 8090 | qwen3-reranker-0.6B-Q8_0 | ~1.3GB | `/v1/rerank` |
|
|
14
|
-
|
|
15
|
-
All three models auto-download via `node-llama-cpp` if no server is running (Metal on Apple Silicon, Vulkan where available, CPU as last resort). Fast with GPU acceleration (Metal/Vulkan); significantly slower on CPU-only.
|
|
16
|
-
|
|
17
|
-
**SOTA upgrade (16GB+ GPU):** CC-BY-NC-4.0 — non-commercial only.
|
|
18
|
-
|
|
19
|
-
| Service | Port | Model | VRAM | Protocol |
|
|
20
|
-
|---|---|---|---|---|
|
|
21
|
-
| Embedding | 8088 | zembed-1-Q4_K_M | ~4.4GB | `/v1/embeddings` |
|
|
22
|
-
| LLM | 8089 | qmd-query-expansion-1.7B-q4_k_m | ~2.2GB | `/v1/chat/completions` |
|
|
23
|
-
| Reranker | 8090 | zerank-2 seq-cls sidecar | ~9GB (bf16) | `/v1/rerank` |
|
|
24
|
-
|
|
25
|
-
zembed-1 (2560d, 32K context, SOTA retrieval) distilled from zerank-2 via zELO — optimal pairing. The SOTA reranker is the **zerank-2 seq-cls sidecar** at `extras/rerankers/zerank-2-seq/` (transformers, bf16, ~9GB VRAM) — **not** a GGUF. The older `zerank-2-Q4_K_M` GGUF is **deprecated**: llama.cpp drops zerank's score head, so its scores are near-zero and uninformative (final ordering stays RRF-dominated). Full SOTA stack ≈ 16GB VRAM.
|
|
26
|
-
|
|
27
|
-
**Remote option:** Set `CLAWMEM_EMBED_URL`, `CLAWMEM_LLM_URL`, `CLAWMEM_RERANK_URL` to the remote host. Set `CLAWMEM_NO_LOCAL_MODELS=true` to prevent surprise fallback downloads.
|
|
28
|
-
|
|
29
|
-
**Cloud embedding:** Set `CLAWMEM_EMBED_API_KEY` + `CLAWMEM_EMBED_URL` + `CLAWMEM_EMBED_MODEL` to use a cloud provider instead of local GPU. Supported: Jina AI (recommended: `jina-embeddings-v5-text-small`, 1024d), OpenAI, Voyage, Cohere. Cloud mode enables batch embedding (50 frags/request), provider-specific retrieval params auto-detected from URL (Jina `task`, Voyage/Cohere `input_type`), server-side truncation, and adaptive TPM-aware pacing. Set `CLAWMEM_EMBED_TPM_LIMIT` to match your tier.
|
|
30
|
-
|
|
31
|
-
**Qwen3 /no_think flag:** Qwen3 uses thinking tokens by default. ClawMem appends `/no_think` to all prompts automatically for structured output.
|
|
32
|
-
|
|
33
|
-
### Model Recommendations
|
|
34
|
-
|
|
35
|
-
| Role | Default (QMD native) | SOTA Upgrade | Notes |
|
|
36
|
-
|---|---|---|---|
|
|
37
|
-
| Embedding | [EmbeddingGemma-300M-Q8_0](https://huggingface.co/ggml-org/embeddinggemma-300M-GGUF) (314MB, 768d) | [zembed-1-Q4_K_M](https://huggingface.co/Abhiray/zembed-1-Q4_K_M-GGUF) (2.4GB, 2560d) | zembed-1: 32K context, SOTA retrieval. `-ub` must match `-b`. |
|
|
38
|
-
| LLM | [qmd-query-expansion-1.7B-q4_k_m](https://huggingface.co/tobil/qmd-query-expansion-1.7B-gguf) (~1.1GB) | Same | QMD's Qwen3-1.7B finetune for query expansion. |
|
|
39
|
-
| Reranker | [qwen3-reranker-0.6B-Q8_0](https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF) (~600MB) | [zerank-2 seq-cls sidecar](extras/rerankers/zerank-2-seq/) (bf16, ~9GB) | zerank-2: NDCG@10 ahead of Cohere rerank-3.5. Served via transformers sidecar, **not** GGUF (the GGUF drops the score head → inert). CC-BY-NC. |
|
|
40
|
-
|
|
41
|
-
### Server Setup (all three use llama-server)
|
|
42
|
-
|
|
43
|
-
```bash
|
|
44
|
-
# === Default (QMD native combo) ===
|
|
45
|
-
|
|
46
|
-
# Embedding (--embeddings flag required)
|
|
47
|
-
llama-server -m embeddinggemma-300M-Q8_0.gguf \
|
|
48
|
-
--embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 2048
|
|
49
|
-
|
|
50
|
-
# LLM (QMD finetuned model)
|
|
51
|
-
llama-server -m qmd-query-expansion-1.7B-q4_k_m.gguf \
|
|
52
|
-
--port 8089 --host 0.0.0.0 -ngl 99 -c 4096 --batch-size 512
|
|
53
|
-
|
|
54
|
-
# Reranker
|
|
55
|
-
llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
|
|
56
|
-
--reranking --port 8090 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 512
|
|
57
|
-
|
|
58
|
-
# === SOTA upgrade (16GB+ GPU) ===
|
|
7
|
+
---
|
|
59
8
|
|
|
60
|
-
|
|
61
|
-
llama-server -m zembed-1-Q4_K_M.gguf \
|
|
62
|
-
--embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
|
|
9
|
+
## Inference at a glance
|
|
63
10
|
|
|
64
|
-
|
|
65
|
-
# cd extras/rerankers/zerank-2-seq && docker compose run --rm convert && docker compose up -d reranker
|
|
66
|
-
# The zerank-2 GGUF is deprecated: llama.cpp drops its score head -> near-zero, uninformative scores.
|
|
67
|
-
```
|
|
11
|
+
Three services — **embedding**, **LLM** (query expansion / intent / A-MEM), **reranker**. Default: all three as `llama-server` with an in-process `node-llama-cpp` fallback that auto-downloads on first use (works with no GPU). The `bin/clawmem` wrapper points at `localhost:8088/8089/8090`. **Always run via `bin/clawmem`** — it sets the endpoints.
|
|
68
12
|
|
|
69
|
-
|
|
13
|
+
**Choose a stack:**
|
|
14
|
+
- **native** (default) — EmbeddingGemma-300M + qmd-query-expansion-1.7B + qwen3-reranker-0.6B · ~4 GB or in-process · **permissive, commercial OK** · zero-config.
|
|
15
|
+
- **z / SOTA** — zembed-1 + qmd-query-expansion-1.7B + zerank-2 seq-cls **sidecar** · ~16 GB · **CC-BY-NC-4.0, non-commercial only** · best recall.
|
|
16
|
+
- **cloud embedding** — Jina/OpenAI/Voyage/Cohere · embedding **only** (LLM + reranker stay local) · no local GPU needed.
|
|
70
17
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
18
|
+
**Landmines:**
|
|
19
|
+
- The zerank-2 **GGUF is inert** — llama.cpp drops the score head → ranking silently collapses to RRF. Use the **seq-cls sidecar**; verify with `clawmem rerank-health` (liveness ≠ correctness).
|
|
20
|
+
- `-ub` must equal `-b` for embedding/reranking (non-causal attention) or `llama-server` asserts.
|
|
21
|
+
- Changing embedding dimensions → `clawmem embed --force` (full re-embed).
|
|
22
|
+
- `CLAWMEM_NO_LOCAL_MODELS=true` to fail fast instead of silent CPU fallback.
|
|
74
23
|
|
|
75
|
-
|
|
76
|
-
curl http://host:8089/v1/models
|
|
24
|
+
→ Stack decision matrix + server setup: [docs/guides/inference-services.md](docs/guides/inference-services.md) · cloud: [docs/guides/cloud-embedding.md](docs/guides/cloud-embedding.md) · all env vars: [docs/reference/configuration.md](docs/reference/configuration.md).
|
|
77
25
|
|
|
78
|
-
|
|
79
|
-
curl http://host:8090/v1/models
|
|
80
|
-
```
|
|
26
|
+
---
|
|
81
27
|
|
|
82
|
-
##
|
|
83
|
-
|
|
84
|
-
| Variable | Default (via wrapper) | Effect |
|
|
85
|
-
|---|---|---|
|
|
86
|
-
| `CLAWMEM_EMBED_URL` | `http://localhost:8088` | Embedding server URL. Local llama-server, cloud API, or falls back to in-process `node-llama-cpp` if unset. |
|
|
87
|
-
| `CLAWMEM_EMBED_API_KEY` | (none) | API key for cloud embedding providers. Sent as Bearer token. Enables cloud mode: skips client-side truncation, sends `truncate: true` + `task` param (LoRA adapter selection for Jina v5), and activates batch embedding with adaptive TPM-aware pacing. |
|
|
88
|
-
| `CLAWMEM_EMBED_MODEL` | `embedding` | Model name for embedding requests. Override for cloud providers (e.g. `jina-embeddings-v5-text-small`). |
|
|
89
|
-
| `CLAWMEM_EMBED_MAX_CHARS` | `6000` | Max chars per embedding input. Default fits EmbeddingGemma (2048 tokens). Set to `1100` for granite-278m (512 tokens). Cloud providers skip truncation. |
|
|
90
|
-
| `CLAWMEM_EMBED_TPM_LIMIT` | `100000` | Tokens-per-minute limit for cloud embedding pacing. Match to your provider tier: Free 100000, Paid 2000000, Premium 50000000. |
|
|
91
|
-
| `CLAWMEM_EMBED_DIMENSIONS` | (none) | Output dimensions for OpenAI `text-embedding-3-*` Matryoshka models (e.g. `512`, `1024`). Only sent when URL contains `openai.com`. |
|
|
92
|
-
| `CLAWMEM_LLM_URL` | `http://localhost:8089` | LLM server for intent, expansion, A-MEM, and entity extraction. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. For better entity extraction quality, point at a 7B+ model or cloud API during `reindex --enrich` (see `docs/internals/entity-resolution.md`). |
|
|
93
|
-
| `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. |
|
|
94
|
-
| `CLAWMEM_NO_LOCAL_MODELS` | `false` | Blocks `node-llama-cpp` from auto-downloading GGUF models. Set `true` for remote-only setups. |
|
|
95
|
-
| `CLAWMEM_VAULTS` | (none) | JSON map of vault name → SQLite path for multi-vault mode. E.g. `{"work":"~/.cache/clawmem/work.sqlite"}` |
|
|
96
|
-
| `CLAWMEM_ENABLE_AMEM` | enabled | A-MEM note construction + link generation during indexing. |
|
|
97
|
-
| `CLAWMEM_ENABLE_CONSOLIDATION` | disabled | Background worker backfills unenriched docs and runs Phase 2/3 consolidation + deductive synthesis. **v0.8.2:** every tick is wrapped in a DB-backed `worker_leases` row (`light-consolidation` key), so multiple host processes against the same vault cannot race on Phase 2 merge writes. Hosted by either `clawmem watch` (canonical, long-lived) or `clawmem mcp` (per-session fallback). |
|
|
98
|
-
| `CLAWMEM_CONSOLIDATION_INTERVAL` | 300000 | Worker interval in ms (min 15000). |
|
|
99
|
-
| `CLAWMEM_HEAVY_LANE` | disabled | **v0.8.0.** Enable the quiet-window heavy maintenance worker — a second, longer-interval consolidation lane with DB-backed `worker_leases` exclusivity, stale-first batching, and `maintenance_runs` journaling. Runs alongside the light lane; off by default. **v0.8.2:** canonical host is `clawmem watch` (e.g. systemd `clawmem-watcher.service`); `clawmem mcp` retains the same gate as a fallback host but emits a stderr warning advising operators to move heavy-lane hosting to the watcher because per-session stdio MCPs may never be alive during the configured quiet window. |
|
|
100
|
-
| `CLAWMEM_HEAVY_LANE_INTERVAL` | 1800000 | **v0.8.0.** Heavy-lane tick interval in ms (min 30000, default 30 min). |
|
|
101
|
-
| `CLAWMEM_HEAVY_LANE_WINDOW_START` | (none) | **v0.8.0.** Start hour (0-23) of the quiet window. Unset → no window. |
|
|
102
|
-
| `CLAWMEM_HEAVY_LANE_WINDOW_END` | (none) | **v0.8.0.** End hour (0-23, exclusive) of the quiet window. Supports midnight wrap (22→6). |
|
|
103
|
-
| `CLAWMEM_HEAVY_LANE_MAX_USAGES` | 30 | **v0.8.0.** Max `context_usage` rows in the last 10 min before the heavy lane skips with `reason='query_rate_high'`. |
|
|
104
|
-
| `CLAWMEM_HEAVY_LANE_OBS_LIMIT` | 100 | **v0.8.0.** Phase 2 stale-first observation batch size. |
|
|
105
|
-
| `CLAWMEM_HEAVY_LANE_DED_LIMIT` | 40 | **v0.8.0.** Phase 3 stale-first deductive candidate batch size. |
|
|
106
|
-
| `CLAWMEM_HEAVY_LANE_SURPRISAL` | `false` | **v0.8.0.** When `true`, the heavy lane seeds Phase 2 with k-NN anomaly-ranked doc ids from `computeSurprisalScores` instead of stale-first ordering. Degrades to stale-first (`surprisal-fallback-stale` metric) on vaults without embeddings. |
|
|
107
|
-
| `CLAWMEM_NUDGE_INTERVAL` | `15` | Prompts between lifecycle tool use before `<vault-nudge>` injection. 0 to disable. |
|
|
108
|
-
| `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 merge-safety score threshold when candidate and existing anchors align. Merges above this normalized 3-gram cosine similarity are allowed. |
|
|
109
|
-
| `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety score threshold (fallback when anchors are ambiguous). |
|
|
110
|
-
| `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration before switching on the gate. |
|
|
111
|
-
| `CLAWMEM_CONTRADICTION_POLICY` | `link` | **v0.7.1.** How the merge-time contradiction gate handles a contradictory merge. `link` keeps both rows active and inserts a `contradicts` edge. `supersede` marks the old row `status='inactive'`. |
|
|
112
|
-
| `CLAWMEM_CONTRADICTION_MIN_CONFIDENCE` | `0.5` | **v0.7.1.** Minimum combined (heuristic + LLM) confidence required before the contradiction gate blocks a merge. Below this, the merge proceeds. |
|
|
113
|
-
|
|
114
|
-
**Note:** The `bin/clawmem` wrapper sets all endpoint defaults. Always use the wrapper — never `bun run src/clawmem.ts` directly. For remote GPU setups, add the same env vars to the watcher service via a systemd drop-in.
|
|
115
|
-
|
|
116
|
-
## Quick Setup
|
|
28
|
+
## Install
|
|
117
29
|
|
|
118
30
|
```bash
|
|
119
|
-
#
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
# Or from source
|
|
123
|
-
git clone https://github.com/yoloshii/clawmem.git ~/clawmem
|
|
124
|
-
cd ~/clawmem && bun install
|
|
125
|
-
ln -sf ~/clawmem/bin/clawmem ~/.bun/bin/clawmem
|
|
126
|
-
|
|
127
|
-
# Bootstrap a vault (init + index + embed + hooks + MCP)
|
|
128
|
-
clawmem bootstrap ~/notes --name notes
|
|
31
|
+
npm install -g clawmem # or: bun add -g clawmem
|
|
32
|
+
clawmem bootstrap ~/notes --name notes # init + index + embed + hooks + MCP
|
|
129
33
|
|
|
130
34
|
# Or step by step:
|
|
131
35
|
clawmem init
|
|
132
36
|
clawmem collection add ~/notes --name notes
|
|
133
37
|
clawmem update --embed
|
|
134
|
-
clawmem setup hooks
|
|
135
|
-
clawmem
|
|
136
|
-
|
|
137
|
-
# Verify
|
|
138
|
-
clawmem doctor # Full health check
|
|
139
|
-
clawmem status # Quick index status
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
### Background Services (systemd user units)
|
|
143
|
-
|
|
144
|
-
The watcher and embed timer keep the vault fresh automatically. Create these after setup:
|
|
145
|
-
|
|
146
|
-
```bash
|
|
147
|
-
# Create systemd user directory
|
|
148
|
-
mkdir -p ~/.config/systemd/user
|
|
149
|
-
|
|
150
|
-
# clawmem-watcher.service — auto-indexes on .md changes
|
|
151
|
-
cat > ~/.config/systemd/user/clawmem-watcher.service << 'EOF'
|
|
152
|
-
[Unit]
|
|
153
|
-
Description=ClawMem file watcher — auto-indexes on .md changes
|
|
154
|
-
After=default.target
|
|
155
|
-
|
|
156
|
-
[Service]
|
|
157
|
-
Type=simple
|
|
158
|
-
ExecStart=%h/clawmem/bin/clawmem watch
|
|
159
|
-
Restart=on-failure
|
|
160
|
-
RestartSec=10
|
|
161
|
-
|
|
162
|
-
[Install]
|
|
163
|
-
WantedBy=default.target
|
|
164
|
-
EOF
|
|
165
|
-
|
|
166
|
-
# clawmem-embed.service — oneshot embedding sweep
|
|
167
|
-
cat > ~/.config/systemd/user/clawmem-embed.service << 'EOF'
|
|
168
|
-
[Unit]
|
|
169
|
-
Description=ClawMem embedding sweep
|
|
170
|
-
|
|
171
|
-
[Service]
|
|
172
|
-
Type=oneshot
|
|
173
|
-
ExecStart=%h/clawmem/bin/clawmem embed
|
|
174
|
-
EOF
|
|
175
|
-
|
|
176
|
-
# clawmem-embed.timer — daily at 04:00
|
|
177
|
-
cat > ~/.config/systemd/user/clawmem-embed.timer << 'EOF'
|
|
178
|
-
[Unit]
|
|
179
|
-
Description=ClawMem daily embedding sweep
|
|
180
|
-
|
|
181
|
-
[Timer]
|
|
182
|
-
OnCalendar=*-*-* 04:00:00
|
|
183
|
-
Persistent=true
|
|
184
|
-
RandomizedDelaySec=300
|
|
185
|
-
|
|
186
|
-
[Install]
|
|
187
|
-
WantedBy=timers.target
|
|
188
|
-
EOF
|
|
189
|
-
|
|
190
|
-
# Enable and start
|
|
191
|
-
systemctl --user daemon-reload
|
|
192
|
-
systemctl --user enable --now clawmem-watcher.service clawmem-embed.timer
|
|
193
|
-
|
|
194
|
-
# Persist across reboots (start without login)
|
|
195
|
-
loginctl enable-linger $(whoami)
|
|
196
|
-
|
|
197
|
-
# Verify
|
|
198
|
-
systemctl --user status clawmem-watcher.service clawmem-embed.timer
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
**Note:** The service files use `%h` (home directory specifier). If clawmem is installed elsewhere, update `ExecStart` paths. For remote GPU setups, add `Environment=CLAWMEM_EMBED_URL=http://host:8088` etc. to both service files (the `bin/clawmem` wrapper sets defaults).
|
|
202
|
-
|
|
203
|
-
---
|
|
204
|
-
|
|
205
|
-
## OpenClaw Integration: Memory System Configuration
|
|
206
|
-
|
|
207
|
-
When using ClawMem with OpenClaw, choose one of two deployment options:
|
|
208
|
-
|
|
209
|
-
**Active Memory coexistence:** ClawMem is fully compatible with OpenClaw's Active Memory plugin (v2026.4.10+). They search different backends (ClawMem vault vs dreaming/wiki) and inject into different prompt regions (user prompt vs system prompt). Both can run simultaneously — no configuration needed.
|
|
210
|
-
|
|
211
|
-
**memory-core dreaming sidecar coexistence (v2026.4.18+, #65411):** ClawMem and `memory-core` dreaming can also run side-by-side. When ClawMem owns the memory slot AND `plugins.entries.memory-core.config.dreaming.enabled = true`, OpenClaw loads `memory-core`'s dreaming engine alongside ClawMem (rest of `memory-core` stays unloaded). Dreaming continues writing to `memory/dreaming/{phase}/YYYY-MM-DD.md` (separate from ClawMem's vault). Default after `openclaw plugins enable clawmem` is `dreaming.enabled = false` (ClawMem-only). Set it `true` if you want to keep the dreaming output stream alongside ClawMem.
|
|
212
|
-
|
|
213
|
-
**OpenClaw v2026.4.11+ required for ClawMem v0.10.0+.** v2026.4.11 tightened plugin discovery (`readdirSync({ withFileTypes: true })` + `dirent.isDirectory()`) and added a plugin-directory ownership check. ClawMem v0.10.0+ ships the new discovery manifest (`src/openclaw/package.json` with `openclaw.extensions`) and defaults to recursive copy (not symlink) installs to clear both gates. Older notes: v2026.4.10 fixed #64192 (`plugins.slots.contextEngine` silently dropped during config normalization, relevant only for pre-v0.10.0 ClawMem that still used the `contextEngine` slot).
|
|
214
|
-
|
|
215
|
-
**ClawMem v0.10.0 uses the `memory` slot, not `contextEngine`.** `openclaw plugins enable clawmem` sets `plugins.slots.memory: "clawmem"` and disables competing memory plugins (`memory-core`, `memory-lancedb`) in one step. The older `openclaw config set plugins.slots.contextEngine clawmem` pattern does not apply to v0.10.0+.
|
|
216
|
-
|
|
217
|
-
### Option 1: ClawMem Exclusive (Recommended)
|
|
218
|
-
|
|
219
|
-
ClawMem handles 100% of structured memory. Disable native memory search (not Active Memory — that's separate and compatible):
|
|
220
|
-
|
|
221
|
-
**Configuration:**
|
|
222
|
-
```bash
|
|
223
|
-
# Disable OpenClaw's native memory search
|
|
224
|
-
openclaw config set agents.defaults.memorySearch.extraPaths "[]"
|
|
225
|
-
|
|
226
|
-
# Verify
|
|
227
|
-
openclaw config get agents.defaults.memorySearch
|
|
228
|
-
# Expected: {"extraPaths": []}
|
|
229
|
-
|
|
230
|
-
# Confirm no native memory index exists
|
|
231
|
-
ls ~/.openclaw/agents/main/memory/
|
|
232
|
-
# Expected: "No such file or directory"
|
|
38
|
+
clawmem setup hooks && clawmem setup mcp
|
|
39
|
+
clawmem doctor # full health check (clawmem status = quick)
|
|
233
40
|
```
|
|
234
41
|
|
|
235
|
-
|
|
236
|
-
- **Tier 2 (90%):** Hooks auto-inject context (session-bootstrap, context-surfacing, staleness-check, decision-extractor, handoff-generator, feedback-loop)
|
|
237
|
-
- **Tier 3 (10%):** Agent-initiated MCP tools (query, intent_search, find_causal_links, etc.)
|
|
238
|
-
|
|
239
|
-
### Option 2: Hybrid (ClawMem + Native)
|
|
240
|
-
|
|
241
|
-
Run both ClawMem and OpenClaw's native memory search for redundancy.
|
|
242
|
-
|
|
243
|
-
**Configuration:**
|
|
244
|
-
```bash
|
|
245
|
-
openclaw config set agents.defaults.memorySearch.extraPaths '["~/documents", "~/notes"]'
|
|
246
|
-
```
|
|
247
|
-
|
|
248
|
-
**Tradeoffs:**
|
|
249
|
-
- Redundant recall from two independent systems
|
|
250
|
-
- 10-15% context window waste from duplicate facts
|
|
251
|
-
- Two memory indices to maintain
|
|
252
|
-
|
|
253
|
-
**Recommendation:** Use Option 1 unless you have a specific need for redundant memory systems.
|
|
42
|
+
→ Quickstart + index patterns: [docs/quickstart.md](docs/quickstart.md) · hooks/MCP: [docs/guides/setup-hooks.md](docs/guides/setup-hooks.md), [docs/guides/setup-mcp.md](docs/guides/setup-mcp.md) · background services: [docs/guides/systemd-services.md](docs/guides/systemd-services.md) · upgrading: [docs/guides/upgrading.md](docs/guides/upgrading.md).
|
|
254
43
|
|
|
255
44
|
---
|
|
256
45
|
|
|
257
|
-
## Memory
|
|
258
|
-
|
|
259
|
-
ClawMem hooks handle ~90% of retrieval automatically. Agent-initiated MCP calls cover the remaining ~10%.
|
|
260
|
-
|
|
261
|
-
### Tier 2 — Automatic (hooks, zero agent effort)
|
|
46
|
+
## Memory retrieval (90/10 rule)
|
|
262
47
|
|
|
263
|
-
|
|
264
|
-
|------|---------|--------|---------|
|
|
265
|
-
| `context-surfacing` | UserPromptSubmit | profile-driven (default 800 + factsTokens sub-budget) | retrieval gate → **multi-turn query construction** (v0.8.1: current prompt + up to 2 recent same-session priors from `context_usage.query_text`, 10-min max age, capped at 2000 chars with current-first preservation — used only for discovery: vector/FTS/expansion, NOT for rerank/scoring/snippet extraction) → **session focus topic resolution** (v0.9.0 §11.4: reads per-session focus file at `~/.cache/clawmem/sessions/<id>.focus`, threaded as intent hint to `expandQuery` + `rerank` + `extractSnippet`) → profile-driven hybrid search (vector if `useVector`, timeout from profile) → FTS supplement → file-aware supplemental search (E13, raw current prompt) → snooze filter → noise filter → spreading activation (E11: co-activated doc boost) → composite scoring → **session focus topic boost** (v0.9.0 §11.4: 1.4× match / 0.75× demote floor 50%, NO-OP on zero matches to preserve baseline ordering) → adaptive threshold → memory type diversification (E10) → tiered injection (HOT/WARM/COLD snippets) → `<vault-context><instruction>…</instruction><facts>…</facts><relationships>…</relationships><vault-facts>…</vault-facts></vault-context>` (v0.7.1: instruction always prepended when context is returned; relationships block lists memory-graph edges where BOTH endpoints are in the surfaced set, truncated first when over budget. **v0.9.0 §11.1:** `<vault-facts>` KG injection block appends raw SPO triple lines from entities seeded by the prompt via three-path prompt-only extraction — canonical IDs + proper nouns + longer-first n-grams — with a dedicated `factsTokens` sub-budget per profile (speed=0 disables the stage, balanced=200, deep=250), cross-entity triple dedup, and truncate-at-triple-boundary budget discipline; fail-open on every error path) + optional `<vault-routing>` hint. Budget, max results, vector timeout, min score, and facts sub-budget all driven by `CLAWMEM_PROFILE`. Raw prompt persisted to `context_usage.query_text` for future multi-turn lookback — except on gated skip paths (slash commands, heartbeats, too-short prompts) where the text is withheld for privacy. |
|
|
266
|
-
| `postcompact-inject` | SessionStart (compact) | 1200 tokens | re-injects authoritative context after compaction: precompact state (600) + recent decisions (400) + antipatterns (150) + vault context (200) → `<vault-postcompact>` |
|
|
267
|
-
| `curator-nudge` | SessionStart | 200 tokens | surfaces curator report actions, nudges when report is stale (>7 days) |
|
|
268
|
-
| `precompact-extract` | PreCompact | — | extracts decisions, file paths, open questions → writes `precompact-state.md` to auto-memory. Query-aware decision ranking. Reindexes auto-memory collection. |
|
|
269
|
-
| `decision-extractor` | Stop | — | LLM extracts observations → `_clawmem/agent/observations/`, infers causal links, detects contradictions, persists observer-emitted SPO triples via `ensureEntityCanonical` (canonical `vault:type:slug` IDs shared with A-MEM) using the tight predicate vocabulary (adopted, migrated_to, deployed_to, runs_on, replaced, depends_on, integrates_with, uses, prefers, avoids, caused_by, resolved_by, owned_by). Eligible observation types: decision/preference/milestone/problem/discovery/feature. Background consolidation worker synthesizes deductive observations from related facts (Phase 3, every ~15 min). |
|
|
270
|
-
| `handoff-generator` | Stop | — | LLM summarizes session → `_clawmem/agent/handoffs/` |
|
|
271
|
-
| `feedback-loop` | Stop | — | tracks referenced notes → boosts confidence, records usage relations + co-activations between co-referenced docs, tracks utility signals (surfaced vs referenced ratio for lifecycle automation), per-turn recall attribution (marks which surfaced docs were cited in which turn) |
|
|
48
|
+
### Tier 2 — automatic (hooks, zero agent effort)
|
|
272
49
|
|
|
273
|
-
|
|
50
|
+
| Hook | Trigger | Does |
|
|
51
|
+
|------|---------|------|
|
|
52
|
+
| `context-surfacing` | UserPromptSubmit | retrieval gate → profile-driven hybrid search → FTS supplement → file-aware search → snooze/noise filters → spreading activation → memory-type diversification → tiered injection → `<vault-context>` (+ optional `<vault-facts>` SPO triples, `<vault-routing>` hint). Budget/results/timeout/threshold driven by `CLAWMEM_PROFILE`. |
|
|
53
|
+
| `postcompact-inject` | SessionStart (compact) | re-injects authoritative state after compaction → `<vault-postcompact>` |
|
|
54
|
+
| `curator-nudge` | SessionStart | surfaces curator actions; nudges when the report is stale |
|
|
55
|
+
| `precompact-extract` | PreCompact | extracts decisions / file paths / open questions before compaction |
|
|
56
|
+
| `decision-extractor` | Stop | LLM → observations + causal links + contradiction detection + SPO triples |
|
|
57
|
+
| `handoff-generator` | Stop | LLM session summary → handoffs |
|
|
58
|
+
| `feedback-loop` | Stop | tracks referenced notes → confidence boosts, co-activations, utility signals |
|
|
274
59
|
|
|
275
|
-
**
|
|
60
|
+
**Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
|
|
61
|
+
**Blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
|
|
62
|
+
**Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) + an activation floor; only `deep` adds query expansion + reranking to the hook path. → [docs/concepts/hooks-vs-mcp.md](docs/concepts/hooks-vs-mcp.md).
|
|
276
63
|
|
|
277
|
-
|
|
64
|
+
### Tier 3 — agent-initiated (one targeted MCP call)
|
|
278
65
|
|
|
279
|
-
|
|
66
|
+
**3-rule escalation gate — escalate ONLY when one fires:**
|
|
67
|
+
1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact the task requires.
|
|
68
|
+
2. **Cross-session question** — "why did we decide X", "what changed since last time", "when did we start Y".
|
|
69
|
+
3. **Pre-irreversible check** — before a destructive / hard-to-reverse change, check the vault for prior decisions.
|
|
280
70
|
|
|
281
|
-
|
|
282
|
-
1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact/chain the task requires. Hooks surface top-k by relevance; if the needed memory wasn't in top-k, escalate.
|
|
283
|
-
2. **Cross-session question** — the task explicitly references prior sessions or decisions: "why did we decide X", "what changed since last time", "when did we start doing Y".
|
|
284
|
-
3. **Pre-irreversible check** — about to make a destructive or hard-to-reverse change (deletion, config change, architecture decision). Check vault for prior decisions before proceeding.
|
|
71
|
+
All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculatively.**
|
|
285
72
|
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
**Once escalated, route by query type:**
|
|
289
|
-
|
|
290
|
-
**PREFERRED:** `memory_retrieve(query)` — auto-classifies and routes to the optimal backend (query, intent_search, session_log, find_similar, or query_plan). Use this instead of manually choosing a tool below.
|
|
73
|
+
**Routing** — PREFERRED: `memory_retrieve(query)` auto-classifies and routes to the optimal backend.
|
|
291
74
|
|
|
292
75
|
```
|
|
293
|
-
1a. General recall
|
|
294
|
-
Full hybrid: BM25 + vector +
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
Optional: candidateLimit=N to tune precision/speed (default 30).
|
|
299
|
-
BM25 strong-signal bypass: skips expansion when top BM25 hit ≥ 0.85 with gap ≥ 0.15 (disabled when intent is provided).
|
|
300
|
-
|
|
301
|
-
1b. Causal/why/when/entity → intent_search(query, enable_graph_traversal=true)
|
|
76
|
+
1a. General recall -> query(query, compact=true, limit=20)
|
|
77
|
+
Full hybrid: BM25 + vector + expansion + deep rerank. Supports compact, collection,
|
|
78
|
+
intent, candidateLimit. BM25 strong-signal bypass skips expansion when top hit >= 0.85
|
|
79
|
+
with gap >= 0.15 (disabled when intent is provided).
|
|
80
|
+
1b. Causal/why/when/entity -> intent_search(query, enable_graph_traversal=true)
|
|
302
81
|
MAGMA intent classification + intent-weighted RRF + multi-hop graph traversal.
|
|
303
|
-
Use DIRECTLY (not as fallback)
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
82
|
+
Use DIRECTLY (not as a fallback) for "why" / "when" / "how did X lead to Y" / entity links.
|
|
83
|
+
Override: force_intent="WHY"|"WHEN"|"ENTITY"|"WHAT". (1a vs 1b are parallel, chosen by query type.)
|
|
84
|
+
1c. Multi-topic -> query_plan(query, compact=true)
|
|
85
|
+
Decomposes into 2-4 typed clauses (bm25/vector/graph), runs them in parallel, merges via RRF.
|
|
86
|
+
2. Progressive disclosure -> multi_get("path1,path2") for full content of top hits
|
|
87
|
+
3. Spot checks -> search(query) (BM25, 0 GPU) or vsearch(query) (vector, 1 GPU)
|
|
88
|
+
4. Chain tracing -> find_causal_links(docid, direction="both", depth=5)
|
|
89
|
+
5. Entity facts -> kg_query(entity) (SPO triples; NOT causal "why" — that's intent_search)
|
|
90
|
+
6. Temporal context -> timeline(docid, before=5, after=5)
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
| Tool | Purpose |
|
|
94
|
+
|------|---------|
|
|
95
|
+
| `memory_retrieve` | **Preferred.** Auto-classifies + routes. Use instead of choosing manually. |
|
|
96
|
+
| `query` | Full hybrid (BM25 + vector + rerank). General-purpose. WRONG for "why" (→ `intent_search`) or cross-session (→ `session_log`). |
|
|
97
|
+
| `intent_search` | "why did we decide X" / "what caused Y" / "who worked on Z". Classifies intent, traverses graph edges — decision chains `query` can't find. |
|
|
98
|
+
| `query_plan` | Multi-topic ("X and also Y", "compare A with B"). Splits + routes each clause. |
|
|
99
|
+
| `search` | BM25 keyword — exact terms, config names, error codes. Fast, 0 GPU. |
|
|
100
|
+
| `vsearch` | Vector semantic — conceptual/fuzzy when vocabulary unknown. ~100ms, 1 GPU. |
|
|
101
|
+
| `get` / `multi_get` | Single doc by path/`#docid` / multiple by glob or comma-list. |
|
|
102
|
+
| `find_similar` | "what else relates to X" — k-NN vector neighbors beyond keyword overlap. |
|
|
103
|
+
| `find_causal_links` | Trace decision chains ("what led to X") over observation docs. |
|
|
104
|
+
| `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why". |
|
|
105
|
+
| `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
|
|
106
|
+
| `profile` | User profile (static facts + dynamic context). |
|
|
107
|
+
| `memory_pin` | +0.3 composite boost. PROACTIVELY for constraints, architecture decisions, corrections. |
|
|
108
|
+
| `memory_snooze` | PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
|
|
109
|
+
| `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
|
|
110
|
+
| `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
|
|
111
|
+
| `timeline` | Temporal neighborhood around a doc. Progressive disclosure: search → timeline → get. |
|
|
112
|
+
| `memory_evolution_status` | How a doc's A-MEM metadata evolved over time. |
|
|
113
|
+
| `lifecycle_status` / `lifecycle_sweep` / `lifecycle_restore` | Lifecycle stats / archive stale (dry-run default) / restore. |
|
|
114
|
+
| `index_stats` / `status` / `reindex` | Doc counts + embedding coverage / quick health / force re-index (does NOT embed). |
|
|
115
|
+
| `beads_sync` / `vault_sync` / `list_vaults` | Beads from Dolt / index a dir into a named vault / list vaults. |
|
|
116
|
+
| `diary_write` / `diary_read` | Diary entries — non-hooked envs only (in Claude Code hooks capture this). |
|
|
117
|
+
|
|
118
|
+
**Multi-vault:** all tools accept an optional `vault` param (omit for single-vault mode). **Progressive disclosure:** ALWAYS `compact=true` first → review snippets/scores → `get` / `multi_get` for full content. → full param docs: [docs/reference/mcp-tools.md](docs/reference/mcp-tools.md).
|
|
325
119
|
|
|
326
|
-
|
|
327
|
-
Structured SPO triples with temporal validity. Different from intent_search:
|
|
328
|
-
- kg_query: "what does ClawMem relate to?" → returns structured facts (subject-predicate-object)
|
|
329
|
-
- intent_search: "why did we choose ClawMem?" → returns documents with causal reasoning
|
|
330
|
-
Use kg_query for entity lookup, intent_search for causal chains.
|
|
331
|
-
|
|
332
|
-
6. Memory debugging → memory_evolution_status(docid)
|
|
333
|
-
|
|
334
|
-
7. Temporal context → timeline(docid, before=5, after=5, same_collection=false)
|
|
335
|
-
Shows what was created/modified before and after a document.
|
|
336
|
-
Use after search to understand chronological neighborhood.
|
|
337
|
-
```
|
|
338
|
-
|
|
339
|
-
**Other tools:**
|
|
340
|
-
- `find_similar(docid)` — "what else relates to X". k-NN vector neighbors — discovers connections beyond keyword overlap.
|
|
341
|
-
- `session_log` — USE THIS for "last time", "yesterday", "what did we do". DO NOT use `query()` for cross-session questions.
|
|
342
|
-
- `profile` — user profile (static facts + dynamic context).
|
|
343
|
-
- `memory_forget(query)` — deactivate a memory by closest match.
|
|
344
|
-
- `memory_pin(query, unpin?)` — +0.3 composite boost. USE PROACTIVELY for constraints, architecture decisions, corrections.
|
|
345
|
-
- `memory_snooze(query, until?)` — USE PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days.
|
|
346
|
-
- `build_graphs(temporal?, semantic?)` — build temporal backbone + semantic graph after bulk ingestion. Not needed after routine indexing (A-MEM handles per-doc links).
|
|
347
|
-
- `beads_sync(project_path?)` — sync Beads issues from Dolt backend (via `bd` CLI) into memory. Usually automatic via watcher.
|
|
348
|
-
- `query_plan(query, compact=true)` — USE THIS for multi-topic queries. `query()` searches as one blob — this splits topics and routes each optimally.
|
|
349
|
-
- `timeline(docid, before=5, after=5, same_collection=false)` — temporal neighborhood around a document. Progressive disclosure: search → timeline → get. Supports same-collection scoping and session correlation.
|
|
350
|
-
- `list_vaults()` — show configured vault names and paths. Empty in single-vault mode (default).
|
|
351
|
-
- `vault_sync(vault, content_root, pattern?, collection_name?)` — index markdown from a directory into a named vault. Restricted-path validation rejects sensitive directories (`/etc/`, `/root/`, `.ssh`, `.env`, `credentials`, etc.).
|
|
352
|
-
- `kg_query(entity, as_of?, direction?)` — query the SPO knowledge graph for an entity's relationships. Returns temporal triples with validity windows. USE THIS for "what does X relate to?", "what was true about X in January?". Uses entity resolution for lookup.
|
|
353
|
-
- `diary_write(entry, topic?, agent?)` — write a diary entry. USE PROACTIVELY in non-hooked environments (Hermes, Gemini, plain MCP) for recording important events and decisions. Do NOT use in Claude Code (hooks handle this automatically).
|
|
354
|
-
- `diary_read(last_n?, agent?)` — read recent diary entries.
|
|
355
|
-
|
|
356
|
-
### Multi-Vault
|
|
357
|
-
|
|
358
|
-
All tools accept an optional `vault` parameter — retrieval (query, search, vsearch, intent_search, memory_retrieve, query_plan), document access (get, multi_get, find_similar, find_causal_links, timeline, memory_evolution_status, session_log), mutations (memory_pin, memory_snooze, memory_forget), lifecycle (lifecycle_status, lifecycle_sweep, lifecycle_restore), and maintenance (status, reindex, index_stats, build_graphs). Omit `vault` for the default vault (single-vault mode). Named vaults are configured in `~/.config/clawmem/config.yaml` under `vaults:` or via `CLAWMEM_VAULTS` env var. Vault paths support `~` expansion.
|
|
359
|
-
|
|
360
|
-
### Memory Lifecycle
|
|
361
|
-
|
|
362
|
-
Pin, snooze, and forget are **manual MCP tools** — not automated. The agent should proactively use them when appropriate:
|
|
363
|
-
|
|
364
|
-
- **Pin** (`memory_pin`) — +0.3 composite boost, ensures persistent surfacing.
|
|
365
|
-
- **Proactive triggers:** User says "remember this" / "don't forget" / "this is important". Architecture or critical design decision just made. User-stated preference or constraint that should persist across sessions.
|
|
366
|
-
- **Do NOT pin:** routine decisions, session-specific context, or observations that will naturally surface via recency.
|
|
367
|
-
- **Snooze** (`memory_snooze`) — temporarily hides from context surfacing until a date.
|
|
368
|
-
- **Proactive triggers:** A memory keeps surfacing but isn't relevant to current work. User says "not now" / "later" / "ignore this for now". Seasonal or time-boxed content (e.g., "revisit after launch").
|
|
369
|
-
- **Forget** (`memory_forget`) — permanently deactivates. Use sparingly.
|
|
370
|
-
- Only when a memory is genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
|
|
371
|
-
- **Contradictions auto-resolve:** When `decision-extractor` detects a new decision contradicting an old one, the old decision's confidence is lowered automatically. No manual intervention needed for superseded decisions. **v0.7.1:** the consolidation worker adds a merge-time contradiction gate — before any Phase 2 merge, it runs a deterministic heuristic + LLM check and either links contradictory observations via a `contradicts` edge (default) or marks the prior row `status='inactive'` (when `CLAWMEM_CONTRADICTION_POLICY=supersede`). Phase 3 deductive synthesis applies the same gate to deductive dedupe matches.
|
|
372
|
-
|
|
373
|
-
### Anti-Patterns
|
|
374
|
-
|
|
375
|
-
- Do NOT manually pick query/intent_search/search when `memory_retrieve` can auto-route.
|
|
376
|
-
- Do NOT call MCP tools every turn — three rules above are the only gates.
|
|
377
|
-
- Do NOT re-search what's already in `<vault-context>`.
|
|
378
|
-
- Do NOT run `status` routinely. Only when retrieval feels broken or after large ingestion.
|
|
379
|
-
- Do NOT pin everything — pin is for persistent high-priority items, not temporary boosting.
|
|
380
|
-
- Do NOT forget memories to "clean up" — let confidence decay and contradiction detection handle it naturally.
|
|
381
|
-
- Do NOT run `build_graphs` after every reindex — A-MEM creates per-doc links automatically. Only after bulk ingestion or when `intent_search` returns weak graph results.
|
|
382
|
-
- Do NOT run `clawmem mine` autonomously — it is a bulk ingestion command (same category as `update`/`reindex`). Suggest it to the user when they mention old conversation exports, but let them run it. Bulk import has disk/embedding cost implications that need user consent. **v0.7.2 adds `--synthesize`** — an opt-in post-import LLM fact extraction pass that turns raw conversation dumps into searchable structured decisions / preferences / milestones / problems with cross-fact relations. Off by default; also requires user consent because it drives additional LLM calls (one per conversation doc). Suggest both together when the user wants to get real value out of old chat exports, not just the raw dumps.
|
|
383
|
-
- Do NOT use `diary_write` in Claude Code — hooks (`decision-extractor`, `handoff-generator`) capture this automatically. Diary is for non-hooked environments only (Hermes, Gemini, plain MCP clients).
|
|
384
|
-
- Do NOT use `kg_query` for causal "why" questions — use `intent_search` or `memory_retrieve`. `kg_query` returns structured entity facts (SPO triples), not reasoning chains.
|
|
385
|
-
|
|
386
|
-
## Tool Selection (one-liner)
|
|
387
|
-
|
|
388
|
-
```
|
|
389
|
-
ClawMem escalation: memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) → multi_get → search/vsearch (spot checks)
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
## Curator Agent
|
|
393
|
-
|
|
394
|
-
Maintenance agent for Tier 3 operations the main agent typically neglects. Install with `clawmem setup curator`.
|
|
395
|
-
|
|
396
|
-
**Invoke:** "curate memory", "run curator", or "memory maintenance"
|
|
397
|
-
|
|
398
|
-
**6 phases:**
|
|
399
|
-
1. Health snapshot — status, index_stats, lifecycle_status, doctor
|
|
400
|
-
2. Lifecycle triage — pin high-value unpinned memories, snooze stale content, propose forget candidates (never auto-confirms)
|
|
401
|
-
3. Retrieval health check — 5 probes (BM25, vector, hybrid, intent/graph, lifecycle)
|
|
402
|
-
4. Maintenance — reflect (cross-session patterns), consolidate --dry-run (dedup candidates)
|
|
403
|
-
5. Graph rebuild — conditional on probe results and embedding state
|
|
404
|
-
6. Collection hygiene — orphan detection, content type distribution
|
|
405
|
-
|
|
406
|
-
**Safety rails:** Never auto-confirms forget. Never runs embed (timer's job). Never modifies config.yaml. All destructive proposals require user approval.
|
|
407
|
-
|
|
408
|
-
## Query Optimization
|
|
409
|
-
|
|
410
|
-
The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector via RRF, and reranks with a cross-encoder. Agents do NOT choose search types — the pipeline handles fusion internally. The optimization levers are: **tool selection**, **query string quality**, **intent**, and **candidateLimit**.
|
|
411
|
-
|
|
412
|
-
### Tool Selection (highest impact)
|
|
413
|
-
|
|
414
|
-
Pick the lightest tool that satisfies the need:
|
|
415
|
-
|
|
416
|
-
| Tool | Cost | When |
|
|
417
|
-
|------|------|------|
|
|
418
|
-
| `search(q, compact=true)` | BM25 only, 0 GPU | Know exact terms, spot-check, fast keyword lookup |
|
|
419
|
-
| `vsearch(q, compact=true)` | Vector only, 1 GPU call | Conceptual/fuzzy, don't know vocabulary |
|
|
420
|
-
| `query(q, compact=true)` | Full hybrid, 3+ GPU calls | General recall, unsure which signal matters |
|
|
421
|
-
| `intent_search(q)` | Hybrid + graph | Why/entity chains (graph traversal), when queries (BM25-biased) |
|
|
422
|
-
| `query_plan(q)` | Hybrid + decomposition | Complex multi-topic queries needing parallel typed retrieval |
|
|
423
|
-
|
|
424
|
-
Use `search` for quick keyword spot-checks. Use `query` for general recall (default Tier 3 workhorse). Use `intent_search` directly (not as fallback) when the question is causal or relational.
|
|
425
|
-
|
|
426
|
-
### Query String Quality
|
|
427
|
-
|
|
428
|
-
The query string feeds BM25 directly (probes first, can short-circuit the pipeline) and anchors the 2×-weighted original signal in RRF.
|
|
429
|
-
|
|
430
|
-
**For keyword recall (BM25):** 2-5 precise terms, no filler. Code identifiers work. BM25 AND's all terms as prefix matches (`perf` matches "performance") — no phrase search or negation syntax. A strong hit (≥ 0.85 with gap ≥ 0.15) skips expansion — faster results.
|
|
431
|
-
|
|
432
|
-
**For semantic recall (vector):** Full natural language question, be specific. `"in the payment service, how are refunds processed"` > `"refunds"`.
|
|
433
|
-
|
|
434
|
-
**Do NOT write hypothetical-answer-style queries.** The expansion LLM already generates hyde variants internally. A long hypothetical dilutes BM25 scoring and duplicates what the pipeline does autonomously.
|
|
435
|
-
|
|
436
|
-
### Intent Parameter
|
|
120
|
+
---
|
|
437
121
|
|
|
438
|
-
|
|
122
|
+
## Query optimization (4 levers)
|
|
439
123
|
|
|
440
|
-
|
|
441
|
-
Do NOT use when: query is already specific, single-domain vault, using `search`/`vsearch` (intent only affects `query`).
|
|
124
|
+
The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector via RRF, and reranks with a cross-encoder — you do NOT choose search types. Your levers:
|
|
442
125
|
|
|
443
|
-
|
|
126
|
+
1. **Tool selection (highest impact)** — pick the lightest tool: `search` (BM25, 0 GPU, exact terms) < `vsearch` (vector, 1 GPU, fuzzy) < `query` (full hybrid) < `intent_search` (why/entity chains) < `query_plan` (multi-topic).
|
|
127
|
+
2. **Query string quality** — keyword recall: 2–5 precise terms, no filler (BM25 ANDs prefix matches, no phrase/negation). Semantic recall: full natural-language question. Do NOT write hypothetical-answer queries (the LLM already generates hyde variants).
|
|
128
|
+
3. **Intent (disambiguation)** — `query("performance", intent="web page load times")` steers 5 stages; use when the term is polysemous or the domain is ambiguous; skip when already specific. Disables BM25 strong-signal bypass (correct — intent signals ambiguity).
|
|
129
|
+
4. **candidateLimit** — RRF candidates reaching the reranker (default 30). Lower (15) for speed/small vault; higher (50) for broad topics/recall.
|
|
444
130
|
|
|
445
|
-
|
|
131
|
+
→ pipeline internals: [docs/internals/query-pipeline.md](docs/internals/query-pipeline.md), [docs/internals/intent-search-pipeline.md](docs/internals/intent-search-pipeline.md). Deeper operating guidance: `SKILL.md`.
|
|
446
132
|
|
|
447
|
-
|
|
133
|
+
---
|
|
448
134
|
|
|
449
|
-
|
|
450
|
-
compositeScore = (0.50 × searchScore + 0.25 × recencyScore + 0.25 × confidenceScore) × qualityMultiplier × coActivationBoost
|
|
451
|
-
```
|
|
135
|
+
## Composite scoring
|
|
452
136
|
|
|
453
|
-
|
|
454
|
-
`coActivationBoost = 1 + min(coCount/10, 0.15)` — documents frequently surfaced together get up to 15% boost.
|
|
455
|
-
Length normalization: `1/(1 + 0.5 × log2(max(bodyLength/500, 1)))` — penalizes verbose entries, floor at 30%.
|
|
456
|
-
Frequency boost: `freqSignal = (revisions-1)×2 + (duplicates-1)`, `freqBoost = min(0.10, log1p(freqSignal)×0.03)`. Revision count weighted 2× vs duplicate count. Capped at 10%.
|
|
457
|
-
Pinned documents get +0.3 additive boost (capped at 1.0).
|
|
137
|
+
Applied automatically to all search results.
|
|
458
138
|
|
|
459
|
-
Recency intent detected ("latest", "recent", "last session"):
|
|
460
139
|
```
|
|
461
|
-
compositeScore = (0.
|
|
140
|
+
compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
|
|
462
141
|
```
|
|
463
142
|
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
| decision, deductive, preference, hub | ∞ | Never decay |
|
|
469
|
-
| antipattern | ∞ | Never decay — accumulated negative patterns persist |
|
|
470
|
-
| project | 120 days | Slow decay |
|
|
471
|
-
| research | 90 days | Moderate decay |
|
|
472
|
-
| problem, milestone, note | 60 days | Default |
|
|
473
|
-
| conversation, progress | 45 days | Faster decay |
|
|
474
|
-
| handoff | 30 days | Fast — recent matters most |
|
|
475
|
-
|
|
476
|
-
Half-lives extend up to 3× for frequently-accessed memories (access reinforcement decays over 90 days).
|
|
477
|
-
Attention decay: non-durable types (handoff, progress, conversation, note, project) lose 5% confidence per week without access. Decision/deductive/preference/hub/research/antipattern are exempt.
|
|
478
|
-
|
|
479
|
-
## Indexing & Graph Building
|
|
143
|
+
- `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× … 1.3×); `coActivationBoost` up to +15%; length-normalized (floor 30%); frequency boost capped +10%; **pinned docs +0.3 additive**.
|
|
144
|
+
- **`query` tool (v0.13.0+):** non-recency queries use **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/context-surfacing keep the 0.50/0.25/0.25 default.
|
|
145
|
+
- **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
|
|
146
|
+
- **Half-lives:** decision/deductive/preference/hub/antipattern = ∞ · project 120d · research 90d · problem/milestone/note 60d · conversation/progress 45d · handoff 30d (extend up to 3× for frequently-accessed).
|
|
480
147
|
|
|
481
|
-
|
|
148
|
+
→ full derivation: [docs/concepts/composite-scoring.md](docs/concepts/composite-scoring.md).
|
|
482
149
|
|
|
483
|
-
|
|
484
|
-
- `**/memory/**/*.md`, `**/memory/**/*.txt` — session logs
|
|
485
|
-
- `**/docs/**/*.md`, `**/docs/**/*.txt` — documentation
|
|
486
|
-
- `**/research/**/*.md`, `**/research/**/*.txt` — research dumps
|
|
487
|
-
- `**/YYYY-MM-DD*.md`, `**/YYYY-MM-DD*.txt` — date-format records
|
|
488
|
-
|
|
489
|
-
### Excluded (even if pattern matches)
|
|
490
|
-
|
|
491
|
-
- `gits/`, `scraped/`, `.git/`, `node_modules/`, `dist/`, `build/`, `vendor/`
|
|
492
|
-
|
|
493
|
-
### Indexing vs Embedding (important distinction)
|
|
494
|
-
|
|
495
|
-
**Infrastructure (Tier 1, no agent action needed):**
|
|
496
|
-
- **`clawmem-watcher`** — keeps index + A-MEM fresh (continuous, on `.md` change). Also watches `.beads/` — routes changes to `syncBeadsIssues()` which queries `bd` CLI for live Dolt data (auto-bridges deps into `memory_relations`). Does NOT embed.
|
|
497
|
-
- **`clawmem-embed` timer** — keeps embeddings fresh (daily). Idempotent, skips already-embedded fragments.
|
|
498
|
-
|
|
499
|
-
**Quality scoring:** Each document gets a `quality_score` (0.0–1.0) computed during indexing based on length, structure (headings, lists), decision keywords, correction keywords, and frontmatter richness. Applied as a multiplier in composite scoring.
|
|
500
|
-
|
|
501
|
-
**Impact of missing embeddings:** `vsearch`, `query` (vector component), `context-surfacing` (vector component), and `generateMemoryLinks()` (neighbor discovery) all depend on embeddings. If embeddings are missing, these degrade silently — BM25 still works, but vector recall and inter-doc link quality suffer.
|
|
502
|
-
|
|
503
|
-
**Agent escape hatches (rare):**
|
|
504
|
-
- `clawmem embed` via CLI if you just wrote a doc and need immediate vector recall in the next turn.
|
|
505
|
-
- Manual `reindex` only when immediate index freshness is required and watcher hasn't caught up.
|
|
506
|
-
|
|
507
|
-
### Graph Population (memory_relations)
|
|
508
|
-
|
|
509
|
-
The `memory_relations` table is populated by multiple independent sources:
|
|
150
|
+
---
|
|
510
151
|
|
|
511
|
-
|
|
512
|
-
|--------|-----------|---------|-------|
|
|
513
|
-
| A-MEM `generateMemoryLinks()` | semantic, supporting, contradicts | Indexing (new docs only) | LLM-assessed confidence + reasoning. Requires embeddings for neighbor discovery. |
|
|
514
|
-
| A-MEM `inferCausalLinks()` | causal | Post-response (IO3 decision-extractor) | Links between `_clawmem/agent/observations/` docs, not arbitrary workspace docs. |
|
|
515
|
-
| Beads `syncBeadsIssues()` | causal, supporting, semantic | `beads_sync` MCP tool or watcher (.beads/ change) | Queries `bd` CLI (Dolt backend). Maps beads deps: blocks→causal, discovered-from→supporting, relates-to→semantic, plus conditional-blocks→causal, caused-by→causal, supersedes→supporting. Metadata: `{origin: "beads"}`. |
|
|
516
|
-
| `buildTemporalBackbone()` | temporal | `build_graphs` MCP tool (manual) | Creation-order edges between all active docs. |
|
|
517
|
-
| `buildSemanticGraph()` | semantic | `build_graphs` MCP tool (manual) | Pure cosine similarity. PK collision: `INSERT OR IGNORE` means A-MEM semantic edges take precedence if they exist first. |
|
|
518
|
-
| Entity co-occurrence graph | entity | A-MEM enrichment (indexing) | LLM entity extraction → quality filters (title/length/blocklist/location validation) → type-agnostic canonical resolution within compatibility buckets (person, org, location, tech=project/service/tool/concept) → `entity_mentions` + `entity_cooccurrences` tables. Entity edges use IDF-based specificity scoring. Feeds ENTITY intent queries and MPFP `[entity, semantic]` patterns. |
|
|
519
|
-
| `consolidated_observations` | supporting, contradicts | Consolidation worker (background, light + heavy lanes) | 3-tier consolidation: facts → observations → mental models. Observations track `proof_count`, `trend` (STABLE/STRENGTHENING/WEAKENING/STALE), and source links. **v0.7.1 safety gates:** name-aware merge gate uses entity-anchor comparison + 3-gram cosine similarity (dual-threshold `CLAWMEM_MERGE_SCORE_NORMAL`=0.93 / `_STRICT`=0.98) to prevent cross-entity merges ("Alice decided X" merging into "Bob decided X"). Merge-time contradiction gate runs deterministic heuristic + LLM check; blocked merges route to `CLAWMEM_CONTRADICTION_POLICY`=`link` (new row + `contradicts` edge, default) or `supersede` (old row `status='inactive'`, new row replaces). **v0.8.0:** the heavy lane calls `consolidateObservations(store, llm, { maxDocs, guarded: true, staleOnly: true })` — `guarded: true` forces merge-safety enforcement regardless of `CLAWMEM_MERGE_GUARD_DRY_RUN`, and `staleOnly: true` reorders candidates by `recall_stats.last_recalled_at ASC` so long-unseen docs bubble up first. Optional `candidateIds` filter plumbs k-NN anomaly ids from `computeSurprisalScores`. |
|
|
520
|
-
| Deductive synthesis | supporting, contradicts | Consolidation worker Phase 3 (background, every ~15 min in the light lane; batched in the heavy lane) | Combines 2-3 related recent observations (decision/preference/milestone/problem, last 7 days) into `content_type='deductive'` documents with `source_doc_ids` provenance. First-class searchable docs with ∞ half-life. **v0.7.1 anti-contamination wrapper:** every draft passes through deterministic pre-checks (empty conclusion, invalid source_indices, pool-only entity contamination via `entity_mentions` or lexical fallback) + LLM validator (fail-open with `validatorFallbackAccepts` counter) + dedupe. Per-reason rejection stats exposed via `DeductiveSynthesisStats` (contaminationRejects, invalidIndexRejects, unsupportedRejects, emptyRejects, dedupSkipped, validatorFallbackAccepts). Contradictory dedupe matches are linked via `contradicts` edges. **v0.8.0:** heavy lane calls `generateDeductiveObservations(store, llm, { maxRecent, guarded: true, staleOnly: true })` for stale-first batching on large vaults. |
|
|
521
|
-
| Heavy maintenance lane journal | — | `clawmem` process with `CLAWMEM_HEAVY_LANE=true` (v0.8.0) | Writes a row to the `maintenance_runs` table for every scheduled heavy-lane attempt (including skips). Columns: `lane` (`heavy`), `phase` (`gate`/`consolidate`/`deductive`), `status` (`started`/`completed`/`failed`/`skipped`), `reason` (`outside_window`/`query_rate_high`/`lease_unavailable`), per-phase `selected`/`processed`/`created`/`null_call` counts, and `metrics_json` with selector type (`stale-first`/`surprisal`/`surprisal-fallback-stale`) + full `DeductiveSynthesisStats` on completed Phase 3 runs. Exclusivity is enforced via the new `worker_leases` table with atomic `INSERT ... ON CONFLICT DO UPDATE ... WHERE expires_at <= ?` acquisition, random 16-byte fencing tokens, and TTL reclaim. Gate uses the existing `context_usage` table (no `query_activity` table) and `recall_stats.last_recalled_at` for stale-first ordering. Off by default — requires explicit opt-in. |
|
|
522
|
-
| Conversation synthesis (`runConversationSynthesis`) | semantic, supporting, contradicts, causal, temporal, entity | `clawmem mine <dir> --synthesize` (opt-in, post-index) | **v0.7.2.** Two-pass LLM pipeline over freshly imported `content_type='conversation'` docs. Pass 1 extracts structured facts (decision/preference/milestone/problem) via `extractFactsFromConversation`, saves each via dedup-aware `saveMemory`, populates a local Set-based alias map. Pass 2 resolves cross-fact links against the local map first (fails closed on ambiguity — multi-candidate titles return unresolved), falls back to collection-scoped SQL lookup with LIMIT 2 ambiguity detection. Relations upsert via `ON CONFLICT DO UPDATE SET weight = MAX(weight, excluded.weight)` — idempotent on equal-weight reruns but monotonically accepts stronger later evidence. Synthesized fact paths are a pure function of `(sourceDocId, slug(title), short sha256(normalizedTitle))`, so reruns update in place instead of creating parallel rows. Counters split into `llmFailures` (null/thrown/invalid-JSON) vs `docsWithNoFacts` (valid-empty extraction). All failures non-fatal — never rolls back the mine import. |
|
|
152
|
+
## Indexing rules
|
|
523
153
|
|
|
524
|
-
**
|
|
154
|
+
- **Indexed** (per collection in `config.yaml`): `**/MEMORY.md` · `**/memory/**` · `**/docs/**` · `**/research/**` · `**/YYYY-MM-DD*` (`.md`/`.txt`).
|
|
155
|
+
- **Excluded (always):** `gits/`, `scraped/`, `.git/`, `node_modules/`, `dist/`, `build/`, `vendor/`. **Never index** credential files (`.env`, `*secrets*`, `*credentials*`) or `gits/`.
|
|
156
|
+
- **Indexing ≠ embedding:** the watcher indexes on `.md` change but does NOT embed; the embed timer (or `clawmem embed`) keeps vectors fresh. Missing embeddings silently degrade vector recall — BM25 still works.
|
|
525
157
|
|
|
526
|
-
|
|
158
|
+
→ architecture + graph building: [docs/concepts/architecture.md](docs/concepts/architecture.md), [docs/internals/graph-traversal.md](docs/internals/graph-traversal.md).
|
|
527
159
|
|
|
528
|
-
|
|
160
|
+
---
|
|
529
161
|
|
|
530
|
-
|
|
162
|
+
## Memory lifecycle (pin / snooze / forget — manual tools)
|
|
531
163
|
|
|
532
|
-
-
|
|
533
|
-
-
|
|
534
|
-
-
|
|
164
|
+
- **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist. Do NOT pin routine/session-specific items.
|
|
165
|
+
- **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
|
|
166
|
+
- **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
|
|
167
|
+
- **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.
|
|
535
168
|
|
|
536
|
-
|
|
169
|
+
---
|
|
537
170
|
|
|
538
|
-
|
|
539
|
-
- When retrieval quality seems degraded — check for unembedded docs or content type distribution issues.
|
|
540
|
-
- Do NOT run routinely.
|
|
171
|
+
## Operational gotchas
|
|
541
172
|
|
|
542
|
-
|
|
173
|
+
- **Empty `context-surfacing`** → prompt < 20 chars, starts with `/`, or nothing scored above threshold. Check `clawmem status` + embedding coverage.
|
|
174
|
+
- **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed`.
|
|
175
|
+
- **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs`. Don't run it after every reindex (A-MEM links per-doc automatically).
|
|
176
|
+
- **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert scores, silently collapsing ranking to RRF.
|
|
177
|
+
- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → almost always the context-surfacing hook's **cold-start** (fresh Bun process + opening a large `index.sqlite` + cold OS page cache), NOT inference. Recurs on memory-constrained hosts (e.g. WSL with a low memory cap) as the cache is evicted between turns. **Durable fix: give the host enough RAM to keep the index cached**; raising the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) is a secondary margin — avoid 15s+ as a standing default. Detail: [docs/troubleshooting.md](docs/troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
|
|
178
|
+
- **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [docs/troubleshooting.md](docs/troubleshooting.md).
|
|
543
179
|
|
|
544
|
-
|
|
180
|
+
---
|
|
545
181
|
|
|
546
|
-
|
|
547
|
-
User Query + optional intent hint
|
|
548
|
-
→ Temporal Extraction (regex date range from query: "last week", "March 2026" → WHERE modified_at BETWEEN filters)
|
|
549
|
-
→ BM25 Probe → Strong Signal Check (skip expansion if top hit ≥ 0.85 with gap ≥ 0.15; disabled when intent provided)
|
|
550
|
-
→ Query Expansion (LLM generates text variants; intent steers expansion prompt)
|
|
551
|
-
→ Parallel (typed routing): BM25(original) + Vector(original) + BM25(lex expansions) + Vector(vec/hyde expansions)
|
|
552
|
-
+ Temporal Proximity (date-range filtered, if temporal constraint extracted)
|
|
553
|
-
+ Entity Graph (conditional 1-hop entity walk from top seeds, if entity signals present)
|
|
554
|
-
→ Original query lists get positional 2× weight in RRF; expanded get 1×
|
|
555
|
-
→ Reciprocal Rank Fusion (k=60, top candidateLimit, up to 4-way parallel legs)
|
|
556
|
-
→ Intent-Aware Chunk Selection (intent terms at 0.5× weight alongside query terms at 1.0×)
|
|
557
|
-
→ Cross-Encoder Reranking (4000 char context; intent prepended to rerank query; chunk dedup; batch cap=4)
|
|
558
|
-
→ Rerank/RRF Blend (blendRerank: 0.9·reranker + 0.1·normalized-RRF tiebreaker; reranker can promote over RRF #1; falls back to RRF order if reranker unavailable)
|
|
559
|
-
→ SAME Composite Scoring
|
|
560
|
-
→ MMR Diversity Filter (Jaccard bigram similarity > 0.6 → demoted, not removed)
|
|
561
|
-
```
|
|
182
|
+
## Anti-patterns
|
|
562
183
|
|
|
563
|
-
|
|
184
|
+
- ❌ Manually pick `query`/`intent_search`/`search` when `memory_retrieve` can auto-route → ✅ `memory_retrieve` first.
|
|
185
|
+
- ❌ Call MCP tools every turn → ✅ only when the 3-rule gate fires.
|
|
186
|
+
- ❌ Re-search what's already in `<vault-context>`.
|
|
187
|
+
- ❌ Run `status` routinely → ✅ only when retrieval feels broken or after large ingestion.
|
|
188
|
+
- ❌ Pin everything → ✅ pin only persistent high-priority items.
|
|
189
|
+
- ❌ Forget memories to "clean up" → ✅ let decay + contradiction detection handle it.
|
|
190
|
+
- ❌ `build_graphs` after every reindex → ✅ only after bulk ingestion or weak graph traversal.
|
|
191
|
+
- ❌ `diary_write` in Claude Code → ✅ hooks capture this (diary is for non-hooked envs only).
|
|
192
|
+
- ❌ `kg_query` for causal "why" → ✅ `intent_search` (kg_query is entity facts, not reasoning chains).
|
|
564
193
|
|
|
565
|
-
|
|
566
|
-
User Query → Intent Classification (WHY/WHEN/ENTITY/WHAT)
|
|
567
|
-
→ BM25 + Vector (intent-weighted RRF: boost BM25 for WHEN, vector for WHY)
|
|
568
|
-
→ Graph Traversal (WHY/ENTITY only; multi-hop beam search over memory_relations)
|
|
569
|
-
Outbound: all edge types (semantic, supporting, contradicts, causal, temporal)
|
|
570
|
-
Inbound: semantic and entity only
|
|
571
|
-
Scores normalized to [0,1] before merge with search results
|
|
572
|
-
→ Cross-Encoder Reranking (200 char context per doc; file-keyed score join)
|
|
573
|
-
→ SAME Composite Scoring (uses stored confidence from contradiction detection + feedback)
|
|
574
|
-
```
|
|
194
|
+
---
|
|
575
195
|
|
|
576
|
-
|
|
196
|
+
## Curator agent
|
|
577
197
|
|
|
578
|
-
|
|
579
|
-
|--------|---------|-----------------|
|
|
580
|
-
| Query expansion | Yes (skipped on strong BM25 signal) | No |
|
|
581
|
-
| Intent hint | Yes (`intent` param steers 5 stages) | Auto-detected (WHY/WHEN/ENTITY/WHAT) |
|
|
582
|
-
| Rerank context | 4000 chars/doc (intent-aware chunk selection) | 200 chars/doc |
|
|
583
|
-
| Chunk dedup | Yes (identical texts share single rerank call) | No |
|
|
584
|
-
| Graph traversal | No | Yes (WHY/ENTITY, multi-hop) |
|
|
585
|
-
| MMR diversity | Yes (`diverse=true` default) | No |
|
|
586
|
-
| `compact` param | Yes | No |
|
|
587
|
-
| `collection` filter | Yes | No |
|
|
588
|
-
| `candidateLimit` | Yes (default 30) | No |
|
|
589
|
-
| Best for | Most queries, progressive disclosure | Causal chains spanning multiple docs |
|
|
198
|
+
Maintenance agent for Tier-3 work the main agent neglects. Install: `clawmem setup curator`. Invoke: **"curate memory" / "run curator" / "memory maintenance"**. Six phases: health snapshot → lifecycle triage (pin/snooze/propose-forget, never auto-confirms) → retrieval health probes → reflect + consolidate `--dry-run` → conditional graph rebuild → collection hygiene. Never auto-confirms forget, never runs embed, never edits config.
|
|
590
199
|
|
|
200
|
+
---
|
|
591
201
|
|
|
592
|
-
##
|
|
202
|
+
## Integrations
|
|
593
203
|
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
Symptom: "[generate] Remote LLM in cooldown, falling back to in-process generation"
|
|
600
|
-
→ Remote LLM server had a transport failure (ECONNREFUSED/ETIMEDOUT). ClawMem set a 60s cooldown
|
|
601
|
-
and is using local node-llama-cpp. Remote will be retried after cooldown expires.
|
|
602
|
-
→ Not an error if you expect local fallback. If you want remote only: ensure llama-server is running,
|
|
603
|
-
or set CLAWMEM_NO_LOCAL_MODELS=true to get null instead of slow local inference.
|
|
604
|
-
|
|
605
|
-
Symptom: Query expansion always fails / returns garbage
|
|
606
|
-
→ On CPU-only systems, in-process inference is significantly slower and less reliable. Systems with GPU acceleration (Metal/Vulkan) handle these models well in-process.
|
|
607
|
-
→ Fix: Run llama-server on a GPU. Even a low-end NVIDIA card handles 1.7B models.
|
|
608
|
-
|
|
609
|
-
Symptom: Vector search returns no results but BM25 works
|
|
610
|
-
→ Missing embeddings. Watcher indexes but does NOT embed.
|
|
611
|
-
→ Fix: Run `clawmem embed` or wait for the daily embed timer.
|
|
612
|
-
|
|
613
|
-
Symptom: llama-server crashes with "non-causal attention requires n_ubatch >= n_tokens"
|
|
614
|
-
→ Embedding/reranking models use non-causal attention. When -b (batch) > -ub (ubatch), the assertion fails.
|
|
615
|
-
→ Fix: Set -ub equal to -b (e.g. -b 2048 -ub 2048). Never omit -ub for embedding/reranking servers.
|
|
616
|
-
|
|
617
|
-
Symptom: context-surfacing hook returns empty
|
|
618
|
-
→ Prompt too short (<20 chars), starts with `/`, or no docs score above threshold.
|
|
619
|
-
→ Fix: Check `clawmem status` for doc counts. Check `clawmem embed` for embedding coverage.
|
|
620
|
-
|
|
621
|
-
Symptom: intent_search returns weak results for WHY/ENTITY
|
|
622
|
-
→ Graph may be sparse (few A-MEM edges).
|
|
623
|
-
→ Fix: Run `build_graphs` to add temporal backbone + semantic edges.
|
|
624
|
-
|
|
625
|
-
Symptom: Watcher logs events but collections show 0 docs after update/reindex
|
|
626
|
-
→ Bun.Glob does not support brace expansion {a,b,c}. Collection patterns returned 0 files.
|
|
627
|
-
→ Fixed 2026-02-12: indexer.ts splits brace patterns into individual Glob scans.
|
|
628
|
-
|
|
629
|
-
Symptom: Watcher fires events but wrong collection processes them (e.g., workspace instead of dharma-propagation)
|
|
630
|
-
→ Collection prefix matching via Array.find() returns first match. Parent paths match before children.
|
|
631
|
-
→ Fixed 2026-02-12: cmdWatch() sorts collections by path length descending (most specific first).
|
|
632
|
-
|
|
633
|
-
Symptom: reindex --force crashes with "UNIQUE constraint failed: documents.collection, documents.path"
|
|
634
|
-
→ Force deactivates rows (active=0) but UNIQUE(collection, path) doesn't discriminate by active flag.
|
|
635
|
-
→ Fixed 2026-02-12: indexer.ts checks for inactive rows and reactivates instead of inserting.
|
|
636
|
-
|
|
637
|
-
Symptom: embed crashes with "UNIQUE constraint failed on vectors_vec primary key" on restart
|
|
638
|
-
→ vectors_vec is a vec0 virtual table — INSERT OR REPLACE is not supported by vec0.
|
|
639
|
-
→ Fixed 2026-03-15: insertEmbedding() uses DELETE (try-catch) + INSERT instead of INSERT OR REPLACE.
|
|
640
|
-
→ Embed can now resume after interrupted runs without --force.
|
|
641
|
-
|
|
642
|
-
Symptom: embed crashes with alternating "no such table: vectors_vec" / "table vectors_vec already exists"
|
|
643
|
-
→ Dimension migration race: --force drops vectors_vec, ensureVecTable per-fragment drops+recreates on dimension
|
|
644
|
-
mismatch, causing rapid table existence flickering between fragments.
|
|
645
|
-
→ Fixed 2026-03-15: ensureVecTable caches verified dimensions (vecTableDims), uses CREATE VIRTUAL TABLE IF NOT EXISTS,
|
|
646
|
-
and clearAllEmbeddings resets the cache. First fragment creates, rest skip the check.
|
|
647
|
-
|
|
648
|
-
Symptom: embed --force with new model produces 3 docs stuck as "Unembedded" but "All documents already embedded"
|
|
649
|
-
→ First fragment (seq=0) failed during a crashed embed run. Later fragments succeeded.
|
|
650
|
-
getHashesNeedingFragments thinks the doc is done but status checks seq=0 specifically.
|
|
651
|
-
→ Fix: Delete partial content_vectors + vectors_vec for the stuck hashes, then re-run embed (no --force).
|
|
652
|
-
The vec0 DELETE try-catch prevents cascading failures during the re-embed.
|
|
653
|
-
|
|
654
|
-
Symptom: reindex --force after v0.2.0 upgrade shows no entity extraction
|
|
655
|
-
→ `reindex --force` treats existing docs as updates (isNew=false). The A-MEM pipeline
|
|
656
|
-
skips entity extraction, link generation, and evolution for updates to avoid churn.
|
|
657
|
-
→ Fix: Use `clawmem reindex --enrich` instead. The `--enrich` flag forces the full
|
|
658
|
-
enrichment pipeline (entity extraction + links + evolution) on all documents.
|
|
659
|
-
→ `--force` alone only refreshes A-MEM notes (keywords, tags, context). `--enrich`
|
|
660
|
-
is needed after major upgrades that add new enrichment stages.
|
|
661
|
-
|
|
662
|
-
Symptom: `clawmem update` crashes with "Binding expected string, TypedArray, boolean, number, bigint or null"
|
|
663
|
-
→ YAML frontmatter values like `title: 2023-09-27` or `title: true` are coerced by gray-matter
|
|
664
|
-
into Date objects or booleans. Bun's SQLite driver rejects these as bind parameters.
|
|
665
|
-
→ Fixed v0.4.2: `parseDocument()` runtime-checks all frontmatter fields via `str()` helper.
|
|
666
|
-
Defense-in-depth `safeTitle` guards in `insertDocument`/`updateDocument`/`reactivateDocument`.
|
|
667
|
-
→ Affects: title, domain, workstream, content_type, review_by — any field gray-matter can coerce.
|
|
668
|
-
|
|
669
|
-
Symptom: CLI reindex/update falls back to node-llama-cpp Vulkan (not GPU server)
|
|
670
|
-
→ GPU env vars only in systemd drop-in, not in wrapper script. CLI invocations missed them.
|
|
671
|
-
→ Fixed 2026-02-12: bin/clawmem wrapper exports CLAWMEM_EMBED_URL/LLM_URL/RERANK_URL defaults.
|
|
672
|
-
→ Always run ClawMem via the `bin/clawmem` wrapper, not `bun run src/clawmem.ts` directly.
|
|
673
|
-
The wrapper sets CLAWMEM_EMBED_URL/LLM_URL/RERANK_URL defaults. Scripts or inline bun
|
|
674
|
-
commands that bypass the wrapper will fall back to in-process node-llama-cpp (slow, CPU).
|
|
675
|
-
|
|
676
|
-
Symptom: "UserPromptSubmit hook error" on context-surfacing hook (intermittent)
|
|
677
|
-
→ SQLite contention between the watcher and the hook. The watcher processes filesystem events
|
|
678
|
-
and holds brief write locks. If the hook fires during a lock, it can exceed its timeout.
|
|
679
|
-
More likely during active conversations with frequent file changes.
|
|
680
|
-
→ v0.1.6 fix: watcher no longer processes session transcript .jsonl files (only .beads/*.jsonl),
|
|
681
|
-
eliminating the most common source of contention.
|
|
682
|
-
→ Default hook timeout is 8s (since v0.1.1). If you have an older install, re-run
|
|
683
|
-
`clawmem setup hooks`. If persistent, restart the watcher: `systemctl --user restart
|
|
684
|
-
clawmem-watcher.service`. Healthy memory is under 100MB — if 400MB+, restart clears it.
|
|
685
|
-
→ v0.2.4 fix: hook's SQLite busy_timeout was 500ms — too tight. During A-MEM enrichment
|
|
686
|
-
or heavy indexing, watcher write locks exceed 500ms, causing SQLITE_BUSY. Raised to
|
|
687
|
-
5000ms (matches MCP server). Still completes within the 8s outer timeout.
|
|
688
|
-
|
|
689
|
-
Symptom: WSL hangs or becomes unresponsive during long sessions / watcher has 100K+ FDs
|
|
690
|
-
→ Pre-v0.2.3: fs.watch(recursive: true) registered inotify watches on EVERY subdirectory,
|
|
691
|
-
including excluded dirs (gits/, node_modules/, .git/). Broad collection paths like
|
|
692
|
-
~/Projects with 67K subdirs exhausted inotify limits.
|
|
693
|
-
→ v0.2.3 fix: watcher walks dir trees at startup, skips excluded subtrees, watches
|
|
694
|
-
non-excluded dirs individually. 500-dir cap per collection path.
|
|
695
|
-
→ Diagnosis: `ls /proc/$(pgrep -f "clawmem.*watch")/fd | wc -l` — healthy < 15K.
|
|
696
|
-
→ If still high: narrow broad collection paths. See docs/troubleshooting.md for details.
|
|
697
|
-
```
|
|
204
|
+
- **Claude Code** — `clawmem setup hooks && clawmem setup mcp`. Hooks = 90% auto; 31 MCP tools = 10%.
|
|
205
|
+
- **OpenClaw** — native memory plugin (`kind: memory`, v0.10.0+): `clawmem setup openclaw`. → [docs/guides/openclaw-plugin.md](docs/guides/openclaw-plugin.md).
|
|
206
|
+
- **Hermes** — `MemoryProvider` plugin: copy `src/hermes/` into `$HERMES_HOME/plugins/clawmem/`. → [docs/guides/hermes-plugin.md](docs/guides/hermes-plugin.md).
|
|
207
|
+
- **REST API** — `clawmem serve [--port 7438]`. → [docs/reference/rest-api.md](docs/reference/rest-api.md).
|
|
698
208
|
|
|
699
|
-
|
|
209
|
+
All integrations share the same SQLite vault — decisions captured in one runtime surface in the others.
|
|
700
210
|
|
|
701
|
-
|
|
211
|
+
---
|
|
702
212
|
|
|
703
|
-
|
|
704
|
-
```bash
|
|
705
|
-
# IO6a: per-prompt context injection (pipe prompt on stdin)
|
|
706
|
-
echo "user query" | clawmem surface --context --stdin
|
|
213
|
+
## Tool selection (one-liner)
|
|
707
214
|
|
|
708
|
-
# IO6b: per-session bootstrap injection (pipe session ID on stdin)
|
|
709
|
-
echo "session-id" | clawmem surface --bootstrap --stdin
|
|
710
215
|
```
|
|
711
|
-
|
|
712
|
-
**Browse and analysis:**
|
|
713
|
-
```bash
|
|
714
|
-
clawmem list [-n N] [-c col] # Browse recent documents (--json for machine output)
|
|
715
|
-
clawmem reflect [N] # Cross-session reflection (last N days, default 14)
|
|
716
|
-
# Shows recurring themes, antipatterns, co-activation clusters
|
|
717
|
-
clawmem consolidate [--dry-run] # Find and archive duplicate low-confidence documents
|
|
718
|
-
# Uses Jaccard similarity within same collection
|
|
216
|
+
memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) -> multi_get -> search/vsearch (spot checks)
|
|
719
217
|
```
|
|
720
218
|
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
```bash
|
|
724
|
-
# Set a focus topic for the current session (multi-word OK)
|
|
725
|
-
clawmem focus set "authentication flow" # uses CLAUDE_SESSION_ID / CLAWMEM_SESSION_ID env var
|
|
726
|
-
clawmem focus set "authentication flow" --session-id abc123 # explicit
|
|
727
|
-
|
|
728
|
-
# Show / clear
|
|
729
|
-
clawmem focus show --session-id abc123
|
|
730
|
-
clawmem focus clear --session-id abc123
|
|
731
|
-
```
|
|
219
|
+
---
|
|
732
220
|
|
|
733
|
-
##
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
750
|
-
|
|
221
|
+
## Deep reference (docs/)
|
|
222
|
+
|
|
223
|
+
| Topic | Path |
|
|
224
|
+
|---|---|
|
|
225
|
+
| Quickstart / introduction | `docs/quickstart.md`, `docs/introduction.md` |
|
|
226
|
+
| Inference stack + server setup | `docs/guides/inference-services.md` |
|
|
227
|
+
| Cloud embedding | `docs/guides/cloud-embedding.md` |
|
|
228
|
+
| Configuration (env vars) | `docs/reference/configuration.md` |
|
|
229
|
+
| Hooks vs MCP | `docs/concepts/hooks-vs-mcp.md` |
|
|
230
|
+
| Composite scoring | `docs/concepts/composite-scoring.md` |
|
|
231
|
+
| Architecture / multi-vault | `docs/concepts/architecture.md`, `docs/concepts/multi-vault.md` |
|
|
232
|
+
| Pipelines / graph / entities | `docs/internals/{query-pipeline,intent-search-pipeline,graph-traversal,entity-resolution}.md` |
|
|
233
|
+
| MCP tools / CLI / REST | `docs/reference/{mcp-tools,cli,rest-api}.md` |
|
|
234
|
+
| Setup (hooks / mcp / systemd) | `docs/guides/{setup-hooks,setup-mcp,systemd-services}.md` |
|
|
235
|
+
| OpenClaw / Hermes plugins | `docs/guides/openclaw-plugin.md`, `docs/guides/hermes-plugin.md` |
|
|
236
|
+
| Troubleshooting | `docs/troubleshooting.md` |
|
|
237
|
+
| Upgrading | `docs/guides/upgrading.md` |
|
|
238
|
+
| Operating ClawMem at query time (portable skill) | `SKILL.md` |
|