clawmem 0.14.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +165 -716
- package/CLAUDE.md +4 -786
- package/README.md +20 -191
- package/SKILL.md +157 -711
- package/docs/clawmem-architecture.excalidraw +2415 -0
- package/docs/clawmem-architecture.png +0 -0
- package/docs/clawmem_hero.jpg +0 -0
- package/docs/concepts/architecture.md +413 -0
- package/docs/concepts/composite-scoring.md +133 -0
- package/docs/concepts/hooks-vs-mcp.md +156 -0
- package/docs/concepts/multi-vault.md +71 -0
- package/docs/contributing.md +101 -0
- package/docs/guides/cloud-embedding.md +134 -0
- package/docs/guides/hermes-plugin.md +187 -0
- package/docs/guides/inference-services.md +144 -0
- package/docs/guides/multi-vault-config.md +84 -0
- package/docs/guides/openclaw-plugin.md +306 -0
- package/docs/guides/setup-hooks.md +146 -0
- package/docs/guides/setup-mcp.md +76 -0
- package/docs/guides/systemd-services.md +332 -0
- package/docs/guides/upgrading.md +566 -0
- package/docs/internals/entity-resolution.md +135 -0
- package/docs/internals/graph-traversal.md +85 -0
- package/docs/internals/intent-search-pipeline.md +103 -0
- package/docs/internals/query-pipeline.md +100 -0
- package/docs/introduction.md +104 -0
- package/docs/quickstart.md +158 -0
- package/docs/reference/cli.md +195 -0
- package/docs/reference/configuration.md +101 -0
- package/docs/reference/mcp-tools.md +336 -0
- package/docs/reference/rest-api.md +204 -0
- package/docs/troubleshooting.md +330 -0
- package/package.json +2 -1
- package/src/memory.ts +2 -0
package/SKILL.md
CHANGED
|
@@ -1,821 +1,267 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: clawmem
|
|
3
|
-
description:
|
|
4
|
-
ClawMem agent reference — detailed operational guidance for the on-device hybrid memory system. Use when: setting up collections/indexing/embedding, troubleshooting retrieval, tuning query optimization (4 levers), understanding pipeline behavior, managing memory lifecycle (pin/snooze/forget), building graphs, or any ClawMem operation beyond basic tool routing.
|
|
3
|
+
description: "ClawMem operational reference for agents at query time — the 3-rule escalation gate, MCP tool routing, the 4 query-optimization levers, pipeline behavior (query vs intent_search), composite scoring, and memory lifecycle (pin/snooze/forget). Use when tuning retrieval, troubleshooting recall quality, or any ClawMem operation beyond the routing already in your global CLAUDE.md / this repo's AGENTS.md. NOT for setup — install / inference-server config / env vars / systemd / indexing config / internals live in AGENTS.md + docs/."
|
|
5
4
|
allowed-tools: "mcp__clawmem__*"
|
|
6
5
|
metadata:
|
|
7
6
|
author: yoloshii
|
|
8
|
-
version:
|
|
7
|
+
version: 2.0.0
|
|
9
8
|
---
|
|
10
9
|
|
|
11
|
-
# ClawMem
|
|
10
|
+
# ClawMem Operational Reference
|
|
12
11
|
|
|
13
|
-
|
|
12
|
+
**Scope: agent-time operations only** — escalation, tool routing, query tuning, pipeline reasoning, composite scoring, lifecycle. Setup, inference-server config, env vars, systemd units, indexing/collection config, graph internals, and the OpenClaw/Hermes plugins are **deliberately not here** — they live in this repo's [`AGENTS.md`](AGENTS.md) + [`docs/`](docs/) (e.g. [`docs/guides/inference-services.md`](docs/guides/inference-services.md), [`docs/reference/configuration.md`](docs/reference/configuration.md), [`docs/troubleshooting.md`](docs/troubleshooting.md), [`docs/internals/`](docs/internals/)). Kept out to avoid drift between this skill and the package.
|
|
14
13
|
|
|
15
|
-
|
|
14
|
+
Routine memory needs neither this skill nor manual MCP calls — **hooks + the ClawMem routing already in `AGENTS.md` / your global `CLAUDE.md` handle ~90%.** Reach for this skill (and Tier-3 tools) only when that isn't enough.
|
|
16
15
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
## Inference Services
|
|
20
|
-
|
|
21
|
-
Three inference services (embedding, LLM, reranker). The **default** stack runs all three as `llama-server` instances; the **SOTA** stack serves the reranker as a transformers sidecar (same `/v1/rerank` contract). The `bin/clawmem` wrapper defaults to `localhost:8088/8089/8090`.
|
|
22
|
-
|
|
23
|
-
**Default (QMD native combo, any GPU or in-process):**
|
|
24
|
-
|
|
25
|
-
| Service | Port | Model | VRAM | Protocol |
|
|
26
|
-
|---|---|---|---|---|
|
|
27
|
-
| Embedding | 8088 | EmbeddingGemma-300M-Q8_0 | ~400MB | `/v1/embeddings` |
|
|
28
|
-
| LLM | 8089 | qmd-query-expansion-1.7B-q4_k_m | ~2.2GB | `/v1/chat/completions` |
|
|
29
|
-
| Reranker | 8090 | qwen3-reranker-0.6B-Q8_0 | ~1.3GB | `/v1/rerank` |
|
|
30
|
-
|
|
31
|
-
All three models auto-download via `node-llama-cpp` if no server is running (Metal on Apple Silicon, Vulkan where available, CPU as last resort). Fast with GPU acceleration (Metal/Vulkan); significantly slower on CPU-only.
|
|
32
|
-
|
|
33
|
-
**SOTA upgrade (16GB+ GPU):** zembed-1-Q4_K_M (embedding, 2560d, ~4.4GB) + the **zerank-2 seq-cls sidecar** (reranker, bf16 ~9GB — see `extras/rerankers/zerank-2-seq/`). Total ~16GB with LLM. Distillation-paired via zELO. zembed-1's `-ub` must match `-b`. **CC-BY-NC-4.0** — non-commercial only. The older `zerank-2-Q4_K_M` GGUF reranker is **deprecated** (llama.cpp drops zerank's score head → inert reranking); use the sidecar.
|
|
34
|
-
|
|
35
|
-
**Remote option:** Set `CLAWMEM_EMBED_URL`, `CLAWMEM_LLM_URL`, `CLAWMEM_RERANK_URL` to remote host. Set `CLAWMEM_NO_LOCAL_MODELS=true` to prevent fallback downloads.
|
|
36
|
-
|
|
37
|
-
**Cloud embedding:** Set `CLAWMEM_EMBED_API_KEY` + `CLAWMEM_EMBED_URL` + `CLAWMEM_EMBED_MODEL` for cloud providers. Supported: Jina AI (`jina-embeddings-v5-text-small`, 1024d), OpenAI, Voyage, Cohere. Batch embedding, TPM-aware pacing, provider-specific params auto-detected.
|
|
38
|
-
|
|
39
|
-
### Server Setup
|
|
40
|
-
|
|
41
|
-
```bash
|
|
42
|
-
# === Default (QMD native combo) ===
|
|
43
|
-
|
|
44
|
-
# Embedding (--embeddings flag required)
|
|
45
|
-
llama-server -m embeddinggemma-300M-Q8_0.gguf \
|
|
46
|
-
--embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 2048
|
|
47
|
-
|
|
48
|
-
# LLM (auto-downloads via node-llama-cpp if no server)
|
|
49
|
-
llama-server -m qmd-query-expansion-1.7B-q4_k_m.gguf \
|
|
50
|
-
--port 8089 --host 0.0.0.0 -ngl 99 -c 4096 --batch-size 512
|
|
51
|
-
|
|
52
|
-
# Reranker (auto-downloads via node-llama-cpp if no server)
|
|
53
|
-
llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
|
|
54
|
-
--reranking --port 8090 --host 0.0.0.0 -ngl 99 -c 2048 --batch-size 512
|
|
55
|
-
|
|
56
|
-
# === SOTA upgrade (16GB+ GPU) ===
|
|
57
|
-
|
|
58
|
-
# Embedding (zembed-1) — -ub must match -b
|
|
59
|
-
llama-server -m zembed-1-Q4_K_M.gguf \
|
|
60
|
-
--embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
|
|
61
|
-
|
|
62
|
-
# Reranker (zerank-2) — seq-cls SIDECAR (transformers, bf16), NOT GGUF:
|
|
63
|
-
# see extras/rerankers/zerank-2-seq/ (the zerank-2 GGUF is deprecated — drops the score head)
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
### Verify Endpoints
|
|
67
|
-
|
|
68
|
-
```bash
|
|
69
|
-
curl http://host:8088/v1/embeddings -d '{"input":"test","model":"embedding"}' -H 'Content-Type: application/json'
|
|
70
|
-
curl http://host:8089/v1/models
|
|
71
|
-
curl http://host:8090/v1/models
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
## Environment Variables
|
|
75
|
-
|
|
76
|
-
| Variable | Default (via wrapper) | Effect |
|
|
77
|
-
|---|---|---|
|
|
78
|
-
| `CLAWMEM_EMBED_URL` | `http://localhost:8088` | Embedding server. Falls back to in-process `node-llama-cpp` if unset. |
|
|
79
|
-
| `CLAWMEM_EMBED_API_KEY` | (none) | API key for cloud embedding. Enables cloud mode: batch embedding, provider-specific params, TPM-aware pacing. |
|
|
80
|
-
| `CLAWMEM_EMBED_MODEL` | `embedding` | Model name for embedding requests. Override for cloud providers (e.g. `jina-embeddings-v5-text-small`). |
|
|
81
|
-
| `CLAWMEM_EMBED_TPM_LIMIT` | `100000` | Tokens-per-minute limit for cloud embedding pacing. Match to your provider tier. |
|
|
82
|
-
| `CLAWMEM_EMBED_DIMENSIONS` | (none) | Output dimensions for OpenAI `text-embedding-3-*` Matryoshka models. |
|
|
83
|
-
| `CLAWMEM_LLM_URL` | `http://localhost:8089` | LLM server. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. |
|
|
84
|
-
| `CLAWMEM_RERANK_URL` | `http://localhost:8090` | Reranker server. Falls to `node-llama-cpp` if unset + `NO_LOCAL_MODELS=false`. |
|
|
85
|
-
| `CLAWMEM_NO_LOCAL_MODELS` | `false` | Blocks `node-llama-cpp` auto-downloads. Set `true` for remote-only. |
|
|
86
|
-
| `CLAWMEM_ENABLE_AMEM` | enabled | A-MEM note construction + link generation during indexing. |
|
|
87
|
-
| `CLAWMEM_ENABLE_CONSOLIDATION` | disabled | Light-lane consolidation worker (Phase 1 backfill + Phase 2 merge + Phase 3 deductive synthesis + Phase 4 recall stats). **v0.8.2:** every tick wraps in a `worker_leases` row (`light-consolidation` key) so multiple host processes against the same vault cannot race on Phase 2 merges. Hosted by `clawmem watch` (canonical) or `clawmem mcp` (per-session fallback). |
|
|
88
|
-
| `CLAWMEM_CONSOLIDATION_INTERVAL` | 300000 | Worker interval in ms (min 15000). |
|
|
89
|
-
| `CLAWMEM_MERGE_SCORE_NORMAL` | `0.93` | **v0.7.1.** Phase 2 merge-safety score threshold when candidate and existing anchors align. |
|
|
90
|
-
| `CLAWMEM_MERGE_SCORE_STRICT` | `0.98` | **v0.7.1.** Strictest merge-safety threshold (fallback when anchors are ambiguous). |
|
|
91
|
-
| `CLAWMEM_MERGE_GUARD_DRY_RUN` | `false` | **v0.7.1.** When `true`, Phase 2 merge-safety rejections are logged but not enforced — use for calibration. |
|
|
92
|
-
| `CLAWMEM_CONTRADICTION_POLICY` | `link` | **v0.7.1.** How the merge-time contradiction gate handles a blocked merge. `link` (default) keeps both rows + inserts `contradicts` edge. `supersede` marks the old row `status='inactive'`. |
|
|
93
|
-
| `CLAWMEM_CONTRADICTION_MIN_CONFIDENCE` | `0.5` | **v0.7.1.** Minimum combined heuristic+LLM confidence required before the contradiction gate blocks a merge. |
|
|
94
|
-
| `CLAWMEM_HEAVY_LANE` | disabled | **v0.8.0.** Enable the quiet-window heavy maintenance worker — a second, longer-interval consolidation lane with DB-backed `worker_leases` exclusivity, stale-first batching, and `maintenance_runs` journaling. Runs alongside the light lane. **v0.8.2:** canonical host is `clawmem watch` (e.g. systemd `clawmem-watcher.service`); `clawmem mcp` retains the same gate as a fallback host but emits a stderr warning advising operators to move heavy-lane hosting to the watcher because per-session stdio MCPs may never be alive during the configured quiet window. |
|
|
95
|
-
| `CLAWMEM_HEAVY_LANE_INTERVAL` | 1800000 | **v0.8.0.** Heavy-lane tick interval in ms (min 30000, default 30 min). |
|
|
96
|
-
| `CLAWMEM_HEAVY_LANE_WINDOW_START` / `_END` | (none) | **v0.8.0.** Start/end hours (0-23) of the quiet window. Supports midnight wrap (22→6). Null on either bound = always in window. |
|
|
97
|
-
| `CLAWMEM_HEAVY_LANE_MAX_USAGES` | 30 | **v0.8.0.** Max `context_usage` rows in the last 10 min before the heavy lane skips with `reason='query_rate_high'`. |
|
|
98
|
-
| `CLAWMEM_HEAVY_LANE_OBS_LIMIT` / `_DED_LIMIT` | 100 / 40 | **v0.8.0.** Phase 2 / Phase 3 stale-first batch sizes. |
|
|
99
|
-
| `CLAWMEM_HEAVY_LANE_SURPRISAL` | `false` | **v0.8.0.** When `true`, the heavy lane seeds Phase 2 with k-NN anomaly-ranked doc ids from `computeSurprisalScores` instead of stale-first ordering. Degrades to stale-first on vaults without embeddings. |
|
|
100
|
-
|
|
101
|
-
**Note:** The `bin/clawmem` wrapper sets all endpoint defaults. Always use the wrapper — never `bun run src/clawmem.ts` directly.
|
|
102
|
-
|
|
103
|
-
---
|
|
104
|
-
|
|
105
|
-
## Quick Setup
|
|
106
|
-
|
|
107
|
-
```bash
|
|
108
|
-
# Install via npm
|
|
109
|
-
bun add -g clawmem # or: npm install -g clawmem
|
|
110
|
-
|
|
111
|
-
# Or from source
|
|
112
|
-
git clone https://github.com/yoloshii/clawmem.git ~/clawmem
|
|
113
|
-
cd ~/clawmem && bun install
|
|
114
|
-
ln -sf ~/clawmem/bin/clawmem ~/.bun/bin/clawmem
|
|
115
|
-
|
|
116
|
-
# Bootstrap a vault (init + index + embed + hooks + MCP)
|
|
117
|
-
clawmem bootstrap ~/notes --name notes
|
|
118
|
-
|
|
119
|
-
# Or step by step:
|
|
120
|
-
clawmem init
|
|
121
|
-
clawmem collection add ~/notes --name notes
|
|
122
|
-
clawmem update --embed
|
|
123
|
-
clawmem setup hooks
|
|
124
|
-
clawmem setup mcp
|
|
125
|
-
|
|
126
|
-
# Verify
|
|
127
|
-
clawmem doctor # Full health check
|
|
128
|
-
clawmem status # Quick index status
|
|
129
|
-
```
|
|
130
|
-
|
|
131
|
-
### Background Services (systemd user units)
|
|
132
|
-
|
|
133
|
-
```bash
|
|
134
|
-
mkdir -p ~/.config/systemd/user
|
|
135
|
-
|
|
136
|
-
# clawmem-watcher.service — auto-indexes on .md changes
|
|
137
|
-
cat > ~/.config/systemd/user/clawmem-watcher.service << 'EOF'
|
|
138
|
-
[Unit]
|
|
139
|
-
Description=ClawMem file watcher — auto-indexes on .md changes
|
|
140
|
-
After=default.target
|
|
141
|
-
|
|
142
|
-
[Service]
|
|
143
|
-
Type=simple
|
|
144
|
-
ExecStart=%h/clawmem/bin/clawmem watch
|
|
145
|
-
Restart=on-failure
|
|
146
|
-
RestartSec=10
|
|
147
|
-
|
|
148
|
-
[Install]
|
|
149
|
-
WantedBy=default.target
|
|
150
|
-
EOF
|
|
151
|
-
|
|
152
|
-
# clawmem-embed.service — oneshot embedding sweep
|
|
153
|
-
cat > ~/.config/systemd/user/clawmem-embed.service << 'EOF'
|
|
154
|
-
[Unit]
|
|
155
|
-
Description=ClawMem embedding sweep
|
|
156
|
-
|
|
157
|
-
[Service]
|
|
158
|
-
Type=oneshot
|
|
159
|
-
ExecStart=%h/clawmem/bin/clawmem embed
|
|
160
|
-
EOF
|
|
161
|
-
|
|
162
|
-
# clawmem-embed.timer — daily at 04:00
|
|
163
|
-
cat > ~/.config/systemd/user/clawmem-embed.timer << 'EOF'
|
|
164
|
-
[Unit]
|
|
165
|
-
Description=ClawMem daily embedding sweep
|
|
166
|
-
|
|
167
|
-
[Timer]
|
|
168
|
-
OnCalendar=*-*-* 04:00:00
|
|
169
|
-
Persistent=true
|
|
170
|
-
RandomizedDelaySec=300
|
|
171
|
-
|
|
172
|
-
[Install]
|
|
173
|
-
WantedBy=timers.target
|
|
174
|
-
EOF
|
|
175
|
-
|
|
176
|
-
# Enable and start
|
|
177
|
-
systemctl --user daemon-reload
|
|
178
|
-
systemctl --user enable --now clawmem-watcher.service clawmem-embed.timer
|
|
179
|
-
loginctl enable-linger $(whoami)
|
|
180
|
-
```
|
|
16
|
+
## Architecture (one-liner)
|
|
181
17
|
|
|
182
|
-
**
|
|
18
|
+
Two tiers: **hooks** = automatic context flow (surfacing, extraction, compaction survival); **MCP tools** = explicit recall / write / lifecycle. Substrate: QMD retrieval (BM25 + vector + RRF + cross-encoder rerank + query expansion), with SAME (composite scoring), MAGMA (intent + graph), and A-MEM (self-evolving notes) layered on top. Do not call standalone QMD tools.
|
|
183
19
|
|
|
184
20
|
---
|
|
185
21
|
|
|
186
|
-
## Tier 2 — Automatic
|
|
22
|
+
## Tier 2 — Automatic retrieval (hooks)
|
|
187
23
|
|
|
188
|
-
Hooks handle ~90% of retrieval
|
|
24
|
+
Hooks handle ~90% of retrieval at zero agent effort.
|
|
189
25
|
|
|
190
|
-
| Hook | Trigger |
|
|
191
|
-
|
|
192
|
-
| `context-surfacing` | UserPromptSubmit |
|
|
193
|
-
| `postcompact-inject` | SessionStart (compact) |
|
|
194
|
-
| `curator-nudge` | SessionStart |
|
|
195
|
-
| `precompact-extract` | PreCompact |
|
|
196
|
-
| `decision-extractor` | Stop |
|
|
197
|
-
| `handoff-generator` | Stop |
|
|
198
|
-
| `feedback-loop` | Stop |
|
|
26
|
+
| Hook | Trigger | Does |
|
|
27
|
+
|------|---------|------|
|
|
28
|
+
| `context-surfacing` | UserPromptSubmit | retrieval gate → profile-driven hybrid search → FTS supplement → file-aware search → snooze/noise filters → spreading activation → memory-type diversification → tiered injection → `<vault-context>` (+ optional `<vault-facts>` / `<vault-routing>`). Budget/results/timeout/threshold driven by `CLAWMEM_PROFILE`. |
|
|
29
|
+
| `postcompact-inject` | SessionStart (compact) | re-injects authoritative state after compaction → `<vault-postcompact>` |
|
|
30
|
+
| `curator-nudge` | SessionStart | surfaces curator actions; nudges when the report is stale |
|
|
31
|
+
| `precompact-extract` | PreCompact | extracts decisions / file paths / open questions before compaction |
|
|
32
|
+
| `decision-extractor` | Stop | LLM → observations + causal links + contradiction detection + SPO triples |
|
|
33
|
+
| `handoff-generator` | Stop | LLM session summary → handoffs |
|
|
34
|
+
| `feedback-loop` | Stop | tracks referenced notes → confidence boosts, co-activations, utility signals |
|
|
199
35
|
|
|
200
|
-
**Default behavior:**
|
|
36
|
+
**Default behavior:** read injected `<vault-context>` first; if sufficient, answer immediately.
|
|
201
37
|
|
|
202
|
-
**Hook blind spots (by design):**
|
|
38
|
+
**Hook blind spots (by design):** hooks filter `_clawmem/` artifacts, enforce score thresholds, and cap token budget — **absence in `<vault-context>` does NOT mean absence in memory.** If expected memory wasn't surfaced, escalate to Tier 3.
|
|
203
39
|
|
|
204
|
-
**
|
|
40
|
+
**Profiles:** `speed` / `balanced` (default) / `deep` set the kept-score ratio (65% / 55% / 45%) and an activation floor. Only `deep` adds query expansion + reranking to the hook path. Profile and the hook `timeout` are set in `~/.claude/settings.json` — see *Operational gotchas* for timeout tuning.
|
|
205
41
|
|
|
206
42
|
---
|
|
207
43
|
|
|
208
|
-
## Tier 3 — Agent-
|
|
44
|
+
## Tier 3 — Agent-initiated retrieval (MCP tools)
|
|
209
45
|
|
|
210
|
-
### 3-
|
|
46
|
+
### 3-rule escalation gate
|
|
211
47
|
|
|
212
48
|
Escalate to MCP tools ONLY when one of these fires:
|
|
213
49
|
|
|
214
|
-
1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact the task requires.
|
|
215
|
-
2. **Cross-session question** —
|
|
216
|
-
3. **Pre-irreversible check** — about to make a destructive
|
|
50
|
+
1. **Low-specificity injection** — `<vault-context>` is empty or lacks the specific fact the task requires.
|
|
51
|
+
2. **Cross-session question** — "why did we decide X", "what changed since last time", "when did we start Y".
|
|
52
|
+
3. **Pre-irreversible check** — about to make a destructive / hard-to-reverse change; check the vault for prior decisions first.
|
|
217
53
|
|
|
218
|
-
All other retrieval is handled by Tier 2 hooks. Do NOT call MCP tools speculatively
|
|
54
|
+
All other retrieval is handled by Tier 2 hooks. **Do NOT call MCP tools speculatively.**
|
|
219
55
|
|
|
220
|
-
### Tool
|
|
56
|
+
### Tool routing
|
|
221
57
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
**PREFERRED:** `memory_retrieve(query)` — auto-classifies and routes to the optimal backend (query, intent_search, session_log, find_similar, or query_plan). Use this instead of manually choosing a tool below.
|
|
58
|
+
**PREFERRED:** `memory_retrieve(query)` — auto-classifies and routes to the optimal backend (query / intent_search / session_log / find_similar / query_plan). Use this instead of manually choosing.
|
|
225
59
|
|
|
226
60
|
```
|
|
227
|
-
1a. General recall
|
|
228
|
-
Full hybrid: BM25 + vector +
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
Optional: candidateLimit=N (default 30).
|
|
232
|
-
BM25 strong-signal bypass: skips expansion when top BM25 >= 0.85 with gap >= 0.15
|
|
233
|
-
(disabled when intent is provided).
|
|
234
|
-
|
|
61
|
+
1a. General recall -> query(query, compact=true, limit=20)
|
|
62
|
+
Full hybrid: BM25 + vector + expansion + deep rerank. Supports compact, collection,
|
|
63
|
+
intent, candidateLimit. BM25 strong-signal bypass skips expansion when top hit >= 0.85
|
|
64
|
+
with gap >= 0.15 (disabled when intent is provided).
|
|
235
65
|
1b. Causal/why/when/entity -> intent_search(query, enable_graph_traversal=true)
|
|
236
66
|
MAGMA intent classification + intent-weighted RRF + multi-hop graph traversal.
|
|
237
|
-
Use DIRECTLY
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
2. Progressive disclosure -> multi_get("path1,path2") for full content of top hits
|
|
249
|
-
|
|
250
|
-
3. Spot checks -> search(query) (BM25, 0 GPU) or vsearch(query) (vector, 1 GPU)
|
|
251
|
-
|
|
252
|
-
4. Chain tracing -> find_causal_links(docid, direction="both", depth=5)
|
|
253
|
-
Traverses causal edges between _clawmem/agent/observations/ docs.
|
|
254
|
-
|
|
255
|
-
5. Entity facts -> kg_query(entity, as_of?, direction?)
|
|
256
|
-
Structured SPO triples with temporal validity. Different from intent_search:
|
|
257
|
-
- kg_query: "what does ClawMem relate to?" -> returns structured facts (subject-predicate-object)
|
|
258
|
-
- intent_search: "why did we choose ClawMem?" -> returns documents with causal reasoning
|
|
259
|
-
Use kg_query for entity lookup, intent_search for causal chains.
|
|
260
|
-
|
|
261
|
-
6. Memory debugging -> memory_evolution_status(docid)
|
|
262
|
-
|
|
263
|
-
7. Temporal context -> timeline(docid, before=5, after=5, same_collection=false)
|
|
264
|
-
Shows what was created/modified before and after a document.
|
|
265
|
-
Use after search to understand chronological neighborhood.
|
|
67
|
+
Use DIRECTLY (not as a fallback) for "why" / "when" / "how did X lead to Y" / entity links.
|
|
68
|
+
Override: force_intent="WHY"|"WHEN"|"ENTITY"|"WHAT".
|
|
69
|
+
(1a vs 1b are parallel options, chosen by query type — not sequential.)
|
|
70
|
+
1c. Multi-topic -> query_plan(query, compact=true)
|
|
71
|
+
Decomposes into 2-4 typed clauses (bm25/vector/graph), runs them in parallel, merges via RRF.
|
|
72
|
+
2. Progressive disclosure -> multi_get("path1,path2") for full content of top hits
|
|
73
|
+
3. Spot checks -> search(query) (BM25, 0 GPU) or vsearch(query) (vector, 1 GPU)
|
|
74
|
+
4. Chain tracing -> find_causal_links(docid, direction="both", depth=5)
|
|
75
|
+
5. Entity facts -> kg_query(entity) (SPO triples; different from intent_search's reasoning chains)
|
|
76
|
+
6. Temporal context -> timeline(docid, before=5, after=5)
|
|
266
77
|
```
|
|
267
78
|
|
|
268
|
-
### All MCP
|
|
79
|
+
### All MCP tools
|
|
269
80
|
|
|
270
81
|
| Tool | Purpose |
|
|
271
82
|
|------|---------|
|
|
272
|
-
| `memory_retrieve` | **Preferred.** Auto-classifies
|
|
273
|
-
| `query` | Full hybrid (BM25 + vector + rerank). General-purpose
|
|
274
|
-
| `intent_search` |
|
|
275
|
-
| `query_plan` |
|
|
276
|
-
| `search` | BM25 keyword —
|
|
277
|
-
| `vsearch` | Vector semantic —
|
|
278
|
-
| `get` |
|
|
279
|
-
| `
|
|
280
|
-
| `
|
|
281
|
-
| `
|
|
282
|
-
| `session_log` |
|
|
83
|
+
| `memory_retrieve` | **Preferred.** Auto-classifies + routes. Use instead of choosing manually. |
|
|
84
|
+
| `query` | Full hybrid (BM25 + vector + rerank). General-purpose. WRONG for "why" (→ `intent_search`) or cross-session (→ `session_log`). |
|
|
85
|
+
| `intent_search` | "why did we decide X" / "what caused Y" / "who worked on Z". Classifies intent, traverses graph edges — returns decision chains `query` can't find. |
|
|
86
|
+
| `query_plan` | Multi-topic queries ("X and also Y", "compare A with B"). Splits + routes each clause. |
|
|
87
|
+
| `search` | BM25 keyword — exact terms, config names, error codes. Fast, 0 GPU. |
|
|
88
|
+
| `vsearch` | Vector semantic — conceptual/fuzzy when vocabulary unknown. ~100ms, 1 GPU. |
|
|
89
|
+
| `get` / `multi_get` | Single doc by path/`#docid` / multiple by glob or comma-list. |
|
|
90
|
+
| `find_similar` | "what else relates to X" — k-NN vector neighbors beyond keyword overlap. |
|
|
91
|
+
| `find_causal_links` | Trace decision chains ("what led to X") over observation docs. |
|
|
92
|
+
| `kg_query` | Entity SPO triples with temporal validity. Entity facts, NOT causal "why" (use `intent_search`). |
|
|
93
|
+
| `session_log` | "last time" / "yesterday" / "what did we do". Do NOT use `query` for cross-session. |
|
|
283
94
|
| `profile` | User profile (static facts + dynamic context). |
|
|
284
|
-
| `
|
|
285
|
-
| `
|
|
286
|
-
| `
|
|
287
|
-
| `build_graphs` |
|
|
288
|
-
| `
|
|
289
|
-
| `
|
|
290
|
-
| `
|
|
291
|
-
| `
|
|
292
|
-
| `
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
| `vault_sync` | Index markdown from a directory into a named vault. Restricted-path validation rejects sensitive directories. |
|
|
296
|
-
| `kg_query` | Query SPO knowledge graph for entity relationships with temporal validity. Accepts entity name or canonical ID (`vault:type:slug`). Triples are populated by decision-extractor from observer-emitted `<triples>` blocks using a canonical predicate vocabulary. |
|
|
297
|
-
| `diary_write` | Write diary entry. Use proactively in non-hooked environments. Do NOT use in Claude Code. |
|
|
298
|
-
| `diary_read` | Read recent diary entries. Filter by agent name. |
|
|
299
|
-
| `lifecycle_status` | Document lifecycle statistics: active, archived, forgotten, pinned, snoozed counts and policy summary. |
|
|
300
|
-
| `lifecycle_sweep` | Run lifecycle policies: archive stale docs. Defaults to dry_run (preview only). |
|
|
301
|
-
| `lifecycle_restore` | Restore auto-archived documents. Filter by query, collection, or all. Does NOT restore manually forgotten docs. |
|
|
302
|
-
|
|
303
|
-
**Multi-vault:** All tools accept an optional `vault` parameter. Omit for the default vault (single-vault mode). Named vaults configured in `~/.config/clawmem/config.yaml` under `vaults:` or via `CLAWMEM_VAULTS` env var. Vault paths support `~` expansion.
|
|
304
|
-
|
|
305
|
-
**Progressive disclosure:** ALWAYS `compact=true` first -> review snippets/scores -> `get(docid)` or `multi_get(pattern)` for full content.
|
|
95
|
+
| `memory_pin` | +0.3 composite boost. Use PROACTIVELY for constraints, architecture decisions, corrections. |
|
|
96
|
+
| `memory_snooze` | Use PROACTIVELY when `<vault-context>` surfaces noise — snooze 30 days. |
|
|
97
|
+
| `memory_forget` | Deactivate a memory by closest match. Sparingly — prefer snooze. |
|
|
98
|
+
| `build_graphs` | Temporal backbone + semantic graph after bulk ingestion. NOT after every reindex. |
|
|
99
|
+
| `timeline` | Temporal neighborhood around a doc. Progressive disclosure: search → timeline → get. |
|
|
100
|
+
| `memory_evolution_status` | How a doc's A-MEM metadata evolved over time. |
|
|
101
|
+
| `lifecycle_status` / `lifecycle_sweep` / `lifecycle_restore` | Lifecycle stats / archive stale (dry-run default) / restore auto-archived. |
|
|
102
|
+
| `index_stats` / `status` / `reindex` | Doc counts + embedding coverage / quick health / force re-index (does NOT embed). |
|
|
103
|
+
| `beads_sync` / `vault_sync` / `list_vaults` | Beads issues from Dolt / index a dir into a named vault / list vaults. |
|
|
104
|
+
|
|
105
|
+
**Multi-vault:** all tools accept an optional `vault` param (omit for single-vault mode). **Progressive disclosure:** ALWAYS `compact=true` first → review snippets/scores → `get` / `multi_get` for full content.
|
|
306
106
|
|
|
307
107
|
---
|
|
308
108
|
|
|
309
|
-
## Query
|
|
109
|
+
## Query optimization (4 levers)
|
|
310
110
|
|
|
311
|
-
|
|
111
|
+
The pipeline autonomously generates lex/vec/hyde variants, fuses BM25 + vector via RRF, and reranks with a cross-encoder — you do NOT choose search types. Your levers are **tool selection, query string quality, intent, and candidateLimit.**
|
|
312
112
|
|
|
313
|
-
### Lever 1
|
|
113
|
+
### Lever 1 — Tool selection (highest impact)
|
|
314
114
|
|
|
315
|
-
Pick the lightest tool that satisfies the need
|
|
115
|
+
Pick the lightest tool that satisfies the need:
|
|
316
116
|
|
|
317
117
|
| Tool | Cost | When |
|
|
318
118
|
|------|------|------|
|
|
319
|
-
| `search(q, compact=true)` | BM25 only, 0 GPU | Know exact terms, spot-check
|
|
320
|
-
| `vsearch(q, compact=true)` | Vector only, 1 GPU
|
|
321
|
-
| `query(q, compact=true)` | Full hybrid, 3+ GPU
|
|
322
|
-
| `intent_search(q)` | Hybrid + graph
|
|
323
|
-
| `query_plan(q, compact=true)` | Hybrid + decomposition | Complex multi-topic
|
|
324
|
-
|
|
325
|
-
Use `search` for quick keyword spot-checks. Use `query` for general recall (default Tier 3 workhorse). Use `intent_search` directly (not as fallback) when the question is causal or relational.
|
|
326
|
-
|
|
327
|
-
### Lever 2: Query String Quality
|
|
328
|
-
|
|
329
|
-
The query string directly feeds BM25 (which probes first and can short-circuit the entire pipeline) and anchors the 2x-weighted original signal in RRF. A good query string is the single biggest determinant of result quality.
|
|
330
|
-
|
|
331
|
-
**For keyword recall (BM25 path):**
|
|
332
|
-
- 2-5 precise terms, no filler words
|
|
333
|
-
- Code identifiers work: `handleError async`
|
|
334
|
-
- BM25 tokenizes on whitespace and separators (`_ - . /` etc., mirroring the index), then AND's all terms as prefix matches (`perf` matches "performance"; `before_compaction` matches docs containing `before` and `compaction`)
|
|
335
|
-
- No phrase search or negation syntax — all terms are positive prefix matches
|
|
336
|
-
- A strong keyword hit (score >= 0.85 with gap >= 0.15) skips expansion entirely — faster results
|
|
337
|
-
|
|
338
|
-
**For semantic recall (vector path):**
|
|
339
|
-
- Full natural language question, be specific
|
|
340
|
-
- Include context: `"in the payment service, how are refunds processed"` > `"refunds"`
|
|
341
|
-
- The expansion LLM generates complementary variants — don't try to do its job
|
|
342
|
-
|
|
343
|
-
**Do NOT write hypothetical-answer-style queries.** The expansion LLM already generates hyde variants internally. Writing a 50-word hypothetical dilutes BM25 scoring and is redundant with what the pipeline does autonomously.
|
|
344
|
-
|
|
345
|
-
### Lever 3: Intent (Disambiguation)
|
|
119
|
+
| `search(q, compact=true)` | BM25 only, 0 GPU | Know exact terms, spot-check |
|
|
120
|
+
| `vsearch(q, compact=true)` | Vector only, 1 GPU | Conceptual/fuzzy, vocabulary unknown |
|
|
121
|
+
| `query(q, compact=true)` | Full hybrid, 3+ GPU | General recall, need best results |
|
|
122
|
+
| `intent_search(q)` | Hybrid + graph | Why/entity chains, when queries |
|
|
123
|
+
| `query_plan(q, compact=true)` | Hybrid + decomposition | Complex multi-topic |
|
|
346
124
|
|
|
347
|
-
|
|
125
|
+
### Lever 2 — Query string quality
|
|
348
126
|
|
|
349
|
-
|
|
350
|
-
query("performance", intent="web page load times and Core Web Vitals")
|
|
351
|
-
```
|
|
352
|
-
|
|
353
|
-
**When to provide:**
|
|
354
|
-
- Query term has multiple meanings in the vault ("performance", "pipeline", "model")
|
|
355
|
-
- You know the domain but the query alone is ambiguous
|
|
356
|
-
- Cross-domain search where same terms appear in different contexts
|
|
127
|
+
The query string feeds BM25 (probes first, can short-circuit the pipeline) and anchors the 2×-weighted original signal in RRF — the single biggest determinant of result quality.
|
|
357
128
|
|
|
358
|
-
**
|
|
359
|
-
-
|
|
360
|
-
-
|
|
361
|
-
- Using `search` or `vsearch` (intent only affects `query` tool)
|
|
129
|
+
- **Keyword recall (BM25):** 2–5 precise terms, no filler. Code identifiers work (`handleError async`). BM25 ANDs all terms as prefix matches (`perf` matches "performance") — no phrase search or negation. A strong hit (≥ 0.85, gap ≥ 0.15) skips expansion.
|
|
130
|
+
- **Semantic recall (vector):** full natural-language question, be specific — `"in the payment service, how are refunds processed"` > `"refunds"`.
|
|
131
|
+
- **Do NOT write hypothetical-answer-style queries** — the expansion LLM already generates hyde variants; a long hypothetical dilutes BM25 and duplicates the pipeline.
|
|
362
132
|
|
|
363
|
-
|
|
133
|
+
### Lever 3 — Intent (disambiguation)
|
|
364
134
|
|
|
365
|
-
|
|
135
|
+
Steers 5 autonomous stages (expansion, reranking, chunk selection, snippet extraction, strong-signal bypass). `query("performance", intent="web page load times and Core Web Vitals")`.
|
|
366
136
|
|
|
367
|
-
|
|
137
|
+
- **Provide when:** the term is polysemous in the vault, or the domain is known but the query alone is ambiguous.
|
|
138
|
+
- **Skip when:** the query is already specific, single-domain vault, or using `search`/`vsearch` (intent only affects `query`).
|
|
139
|
+
- Intent disables the BM25 strong-signal bypass (forces full expansion+rerank) — correct, since intent signals ambiguity.
|
|
368
140
|
|
|
369
|
-
|
|
370
|
-
query("architecture decisions", candidateLimit=15) # Faster, more precise
|
|
371
|
-
query("architecture decisions", candidateLimit=50) # Broader recall, slower
|
|
372
|
-
```
|
|
141
|
+
### Lever 4 — candidateLimit
|
|
373
142
|
|
|
374
|
-
Lower
|
|
375
|
-
Higher when: broad topic, vault is large, recall matters more than speed.
|
|
143
|
+
How many RRF candidates reach the cross-encoder reranker (default 30). Lower (`15`) for high-confidence/speed/small-vault; higher (`50`) for broad topics/large vault/recall-over-speed.
|
|
376
144
|
|
|
377
145
|
---
|
|
378
146
|
|
|
379
|
-
## Pipeline
|
|
147
|
+
## Pipeline behavior
|
|
380
148
|
|
|
381
149
|
### `query` (default Tier 3 workhorse)
|
|
382
150
|
|
|
383
151
|
```
|
|
384
|
-
|
|
385
|
-
->
|
|
386
|
-
->
|
|
387
|
-
->
|
|
388
|
-
->
|
|
389
|
-
->
|
|
390
|
-
-> Intent-
|
|
391
|
-
->
|
|
392
|
-
->
|
|
393
|
-
-> Composite Scoring
|
|
394
|
-
-> MMR Diversity Filter (Jaccard bigram similarity > 0.6 -> demoted, not removed)
|
|
152
|
+
Query + optional intent
|
|
153
|
+
-> Temporal extraction (date ranges from "last week"/"March 2026")
|
|
154
|
+
-> BM25 probe -> strong-signal check (skip expansion if top >= 0.85, gap >= 0.15; off when intent given)
|
|
155
|
+
-> Query expansion (LLM text variants; intent steers the prompt)
|
|
156
|
+
-> Parallel typed legs: BM25(orig) + Vector(orig) + BM25(lex exp) + Vector(vec/hyde exp) [+ temporal/entity if signalled]
|
|
157
|
+
-> RRF (k=60; original lists get 2x positional weight, expanded 1x; top candidateLimit)
|
|
158
|
+
-> Intent-aware chunk selection -> cross-encoder rerank (4000-char ctx; chunk dedup)
|
|
159
|
+
-> rerank/RRF blend (0.9 reranker + 0.1 RRF tiebreaker; falls back to RRF if reranker down)
|
|
160
|
+
-> composite scoring -> MMR diversity (Jaccard bigram > 0.6 demoted, not removed)
|
|
395
161
|
```
|
|
396
162
|
|
|
397
163
|
### `intent_search` (specialist for causal chains)
|
|
398
164
|
|
|
399
165
|
```
|
|
400
|
-
|
|
401
|
-
-> BM25 + Vector (intent-weighted RRF:
|
|
402
|
-
-> Graph
|
|
403
|
-
|
|
404
|
-
Inbound: semantic and entity only
|
|
405
|
-
Scores normalized to [0,1] before merge with search results
|
|
406
|
-
-> Cross-Encoder Reranking (200 char context per doc; file-keyed score join)
|
|
407
|
-
-> Composite Scoring (uses stored confidence from contradiction detection + feedback)
|
|
166
|
+
Query -> intent classification (WHY/WHEN/ENTITY/WHAT)
|
|
167
|
+
-> BM25 + Vector (intent-weighted RRF: BM25 for WHEN, vector for WHY)
|
|
168
|
+
-> Graph traversal (WHY/ENTITY; multi-hop over memory_relations; outbound all edge types, inbound semantic+entity)
|
|
169
|
+
-> cross-encoder rerank (200-char ctx) -> composite scoring
|
|
408
170
|
```
|
|
409
171
|
|
|
410
|
-
|
|
172
|
+
**MPFP fusion is max-score, NOT RRF.** The graph stage runs meta-path patterns (`[semantic,causal]`, `[entity,temporal]`, …) via Forward Push (α=0.15) and fuses by max-score ("best supporting path wins"), because propagation magnitude carries signal. This is distinct from the *outer* retrieval, which DOES fuse BM25+vector via RRF — two layers, two fusion rules, by design.
|
|
173
|
+
|
|
174
|
+
### Key differences
|
|
411
175
|
|
|
412
176
|
| Aspect | `query` | `intent_search` |
|
|
413
177
|
|--------|---------|-----------------|
|
|
414
|
-
| Query expansion | Yes (skipped on strong BM25
|
|
415
|
-
| Intent
|
|
416
|
-
| Rerank context | 4000 chars/doc
|
|
417
|
-
| Chunk dedup | Yes (identical texts share single rerank call) | No |
|
|
178
|
+
| Query expansion | Yes (skipped on strong BM25) | No |
|
|
179
|
+
| Intent | `intent` param steers 5 stages | Auto-detected (WHY/WHEN/ENTITY/WHAT) |
|
|
180
|
+
| Rerank context | 4000 chars/doc | 200 chars/doc |
|
|
418
181
|
| Graph traversal | No | Yes (WHY/ENTITY, multi-hop) |
|
|
419
|
-
| MMR diversity | Yes
|
|
420
|
-
| `compact`
|
|
421
|
-
|
|
|
422
|
-
| `candidateLimit` | Yes (default 30) | No |
|
|
423
|
-
| Best for | Most queries, progressive disclosure | Causal chains spanning multiple docs |
|
|
424
|
-
|
|
425
|
-
### `intent_search` force_intent Guide
|
|
426
|
-
|
|
427
|
-
| Override | Triggers |
|
|
428
|
-
|----------|----------|
|
|
429
|
-
| `WHY` | "why", "what led to", "rationale", "tradeoff", "decision behind" |
|
|
430
|
-
| `ENTITY` | Named component/person/service needing cross-doc linkage |
|
|
431
|
-
| `WHEN` | Timelines, first/last occurrence, "when did this change/regress" |
|
|
182
|
+
| MMR diversity | Yes | No |
|
|
183
|
+
| `compact` / `collection` / `candidateLimit` | Yes | No |
|
|
184
|
+
| Best for | most queries, progressive disclosure | causal chains across docs |
|
|
432
185
|
|
|
433
|
-
**WHEN
|
|
186
|
+
**force_intent:** `WHY` ("why", "what led to", "rationale", "tradeoff") · `ENTITY` (named component/person/service needing cross-doc linkage) · `WHEN` (timelines, first/last, "when did this change") — for WHEN start with `enable_graph_traversal=false`, fall back to `query()` if recall drifts.
|
|
434
187
|
|
|
435
188
|
---
|
|
436
189
|
|
|
437
|
-
## Composite
|
|
438
|
-
|
|
439
|
-
Applied automatically to all search tool results.
|
|
440
|
-
|
|
441
|
-
```
|
|
442
|
-
compositeScore = (0.50 x searchScore + 0.25 x recencyScore + 0.25 x confidenceScore) x qualityMultiplier x coActivationBoost
|
|
443
|
-
```
|
|
444
|
-
|
|
445
|
-
Where `qualityMultiplier = 0.7 + 0.6 x qualityScore` (range: 0.7x penalty to 1.3x boost).
|
|
446
|
-
`coActivationBoost = 1 + min(coCount/10, 0.15)` — documents frequently surfaced together get up to 15% boost.
|
|
447
|
-
|
|
448
|
-
Length normalization: `1/(1 + 0.5 x log2(max(bodyLength/500, 1)))` — penalizes verbose entries, floor at 30%.
|
|
449
|
-
|
|
450
|
-
Frequency boost: `freqSignal = (revisions-1)x2 + (duplicates-1)`, `freqBoost = min(0.10, log1p(freqSignal)x0.03)`. Revision count weighted 2x vs duplicate count. Capped at 10%.
|
|
190
|
+
## Composite scoring (how ranking works)
|
|
451
191
|
|
|
452
|
-
|
|
192
|
+
Applied automatically to all search results.
|
|
453
193
|
|
|
454
|
-
### Recency Intent Detected ("latest", "recent", "last session")
|
|
455
|
-
|
|
456
|
-
```
|
|
457
|
-
compositeScore = (0.10 x searchScore + 0.70 x recencyScore + 0.20 x confidenceScore) x qualityMultiplier x coActivationBoost
|
|
458
194
|
```
|
|
459
|
-
|
|
460
|
-
### Content Type Half-Lives
|
|
461
|
-
|
|
462
|
-
| Content Type | Half-Life | Effect |
|
|
463
|
-
|--------------|-----------|--------|
|
|
464
|
-
| decision, deductive, preference, hub | infinity | Never decay |
|
|
465
|
-
| antipattern | infinity | Never decay — accumulated negative patterns persist |
|
|
466
|
-
| project | 120 days | Slow decay |
|
|
467
|
-
| research | 90 days | Moderate decay |
|
|
468
|
-
| problem, milestone, note | 60 days | Default |
|
|
469
|
-
| conversation, progress | 45 days | Faster decay |
|
|
470
|
-
| handoff | 30 days | Fast — recent matters most |
|
|
471
|
-
|
|
472
|
-
Half-lives extend up to 3x for frequently-accessed memories (access reinforcement decays over 90 days).
|
|
473
|
-
|
|
474
|
-
Attention decay: non-durable types (handoff, progress, note, project) lose 5% confidence per week without access. Decision/hub/research/antipattern are exempt.
|
|
475
|
-
|
|
476
|
-
---
|
|
477
|
-
|
|
478
|
-
## Indexing & Graph Building
|
|
479
|
-
|
|
480
|
-
### What Gets Indexed (per collection in config.yaml)
|
|
481
|
-
|
|
482
|
-
- `**/MEMORY.md` — any depth
|
|
483
|
-
- `**/memory/**/*.md`, `**/memory/**/*.txt` — session logs
|
|
484
|
-
- `**/docs/**/*.md`, `**/docs/**/*.txt` — documentation
|
|
485
|
-
- `**/research/**/*.md`, `**/research/**/*.txt` — research dumps
|
|
486
|
-
- `**/YYYY-MM-DD*.md`, `**/YYYY-MM-DD*.txt` — date-format records
|
|
487
|
-
|
|
488
|
-
### Excluded (even if pattern matches)
|
|
489
|
-
|
|
490
|
-
- `gits/`, `scraped/`, `.git/`, `node_modules/`, `dist/`, `build/`, `vendor/`
|
|
491
|
-
|
|
492
|
-
### Indexing vs Embedding
|
|
493
|
-
|
|
494
|
-
**Infrastructure (Tier 1, no agent action):**
|
|
495
|
-
- **`clawmem-watcher`** — keeps index + A-MEM fresh (continuous, on `.md` change). Watches `.beads/` too. Does NOT embed.
|
|
496
|
-
- **`clawmem-embed` timer** — keeps embeddings fresh (daily). Idempotent, skips already-embedded fragments.
|
|
497
|
-
|
|
498
|
-
**Quality scoring:** Each document gets `quality_score` (0.0-1.0) during indexing based on length, structure (headings, lists), decision keywords, correction keywords, frontmatter richness. Applied as multiplier in composite scoring.
|
|
499
|
-
|
|
500
|
-
**Impact of missing embeddings:** `vsearch`, `query` (vector component), `context-surfacing` (vector component), and `generateMemoryLinks()` all depend on embeddings. BM25 still works, but vector recall and inter-doc link quality suffer.
|
|
501
|
-
|
|
502
|
-
**Agent escape hatches (rare):**
|
|
503
|
-
- `clawmem embed` via CLI for immediate vector recall after writing a doc.
|
|
504
|
-
- Manual `reindex` only when watcher hasn't caught up.
|
|
505
|
-
|
|
506
|
-
### Adding New Collections
|
|
507
|
-
|
|
508
|
-
```bash
|
|
509
|
-
# 1. Edit config
|
|
510
|
-
Edit ~/.config/clawmem/config.yaml
|
|
511
|
-
|
|
512
|
-
# 2. Reindex (BM25 only)
|
|
513
|
-
mcp__clawmem__reindex()
|
|
514
|
-
|
|
515
|
-
# 3. Embed (vectors, CLI only)
|
|
516
|
-
CLAWMEM_PATH=~/clawmem ~/clawmem/bin/clawmem embed
|
|
517
|
-
|
|
518
|
-
# 4. Verify
|
|
519
|
-
mcp__clawmem__search(query, collection="name", compact=true) # BM25
|
|
520
|
-
mcp__clawmem__vsearch(query, collection="name", compact=true) # vector
|
|
195
|
+
compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore) × qualityMultiplier × coActivationBoost
|
|
521
196
|
```
|
|
522
197
|
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
| A-MEM `generateMemoryLinks()` | semantic, supporting, contradicts | Indexing (new docs) | LLM-assessed confidence. Requires embeddings. |
|
|
530
|
-
| A-MEM `inferCausalLinks()` | causal | Post-response (decision-extractor) | Links `_clawmem/agent/observations/` docs only. |
|
|
531
|
-
| Beads `syncBeadsIssues()` | causal, supporting, semantic | `beads_sync` MCP or watcher | Queries `bd` CLI (Dolt backend). |
|
|
532
|
-
| `buildTemporalBackbone()` | temporal | `build_graphs` MCP (manual) | Creation-order edges. |
|
|
533
|
-
| `buildSemanticGraph()` | semantic | `build_graphs` MCP (manual) | Pure cosine similarity. A-MEM edges take precedence (first-writer wins). |
|
|
534
|
-
| `consolidated_observations` | supporting, contradicts | Consolidation worker (background, light + heavy lanes) | **v0.7.1 safety gates:** Phase 2 name-aware merge gate (entity anchors + 3-gram cosine, dual-threshold `CLAWMEM_MERGE_SCORE_NORMAL`=0.93 / `_STRICT`=0.98) blocks cross-entity merges. Merge-time contradiction gate (heuristic + LLM) routes blocked merges to `link` (default, inserts `contradicts` edge) or `supersede` (old row `status='inactive'`) via `CLAWMEM_CONTRADICTION_POLICY`. **v0.8.0:** heavy lane calls `consolidateObservations(store, llm, { maxDocs, guarded: true, staleOnly: true, candidateIds? })` — `guarded: true` forces enforcement regardless of `CLAWMEM_MERGE_GUARD_DRY_RUN`, `staleOnly: true` reorders by `recall_stats.last_recalled_at ASC`, and optional `candidateIds` plumbs surprisal-selector output. |
|
|
535
|
-
| Deductive synthesis | supporting, contradicts | Consolidation worker Phase 3 (every ~15 min in light lane; batched in heavy lane) | Combines 2-3 related observations (decision/preference/milestone/problem, last 7 days) into `content_type='deductive'` docs. **v0.7.1 anti-contamination:** deterministic pre-checks (empty/invalid_indices/pool-only entity contamination) + LLM validator (fail-open, `validatorFallbackAccepts` counter) + dedupe. Per-reason rejection stats via `DeductiveSynthesisStats`. Contradictory dedupe matches linked via `contradicts` edges. **v0.8.0:** heavy lane calls `generateDeductiveObservations(store, llm, { maxRecent, guarded: true, staleOnly: true })` for stale-first batching. |
|
|
536
|
-
| Conversation synthesis | semantic, supporting, contradicts, causal, temporal, entity | `clawmem mine <dir> --synthesize` (opt-in, post-index) | **v0.7.2.** Two-pass LLM pipeline over freshly imported `content_type='conversation'` docs. Pass 1 extracts structured decision/preference/milestone/problem facts + aliases + cross-fact links, saves via dedup-aware `saveMemory`, populates ambiguity-aware local Set map. Pass 2 resolves links (local first, SQL fallback with `LIMIT 2` ambiguity detection), upserts relations via `ON CONFLICT DO UPDATE SET weight = MAX(weight, excluded.weight)`. Synthesized paths are a pure function of `(sourceDocId, slug(title), short sha256(normalizedTitle))` so reruns update in place. All failures non-fatal. Counters split: `llmFailures` (LLM/parse error) vs `docsWithNoFacts` (valid empty extraction). |
|
|
537
|
-
| Heavy maintenance lane journal | — | `CLAWMEM_HEAVY_LANE=true` (v0.8.0) | Writes a row to `maintenance_runs` for every scheduled heavy-lane attempt (including skips). Columns: `lane` (`heavy`), `phase` (`gate`/`consolidate`/`deductive`), `status` (`started`/`completed`/`failed`/`skipped`), `reason` (`outside_window`/`query_rate_high`/`lease_unavailable`), per-phase counts, and `metrics_json` with selector type (`stale-first`/`surprisal`/`surprisal-fallback-stale`) + full `DeductiveSynthesisStats`. Exclusivity via new `worker_leases` table (atomic `INSERT ... ON CONFLICT DO UPDATE ... WHERE expires_at <= ?`, 16-byte fencing tokens, TTL reclaim). Gate reuses `context_usage` (no `query_activity` table). |
|
|
538
|
-
|
|
539
|
-
**Graph traversal asymmetry:** `adaptiveTraversal()` traverses all edge types outbound (source->target) but only `semantic` and `entity` inbound.
|
|
540
|
-
|
|
541
|
-
### When to Run `build_graphs`
|
|
542
|
-
|
|
543
|
-
- After **bulk ingestion** — adds temporal backbone + semantic gap filling.
|
|
544
|
-
- When `intent_search` for WHY/ENTITY returns weak results and you suspect graph sparsity.
|
|
545
|
-
- Do NOT run after every reindex (A-MEM handles per-doc links automatically).
|
|
546
|
-
|
|
547
|
-
---
|
|
548
|
-
|
|
549
|
-
## Memory Lifecycle
|
|
550
|
-
|
|
551
|
-
Pin, snooze, and forget are **manual MCP tools**.
|
|
552
|
-
|
|
553
|
-
### Pin (`memory_pin`)
|
|
554
|
-
|
|
555
|
-
+0.3 composite boost, ensures persistent surfacing.
|
|
556
|
-
|
|
557
|
-
**Proactive triggers:**
|
|
558
|
-
- User says "remember this" / "don't forget" / "this is important"
|
|
559
|
-
- Architecture or critical design decision just made
|
|
560
|
-
- User-stated preference or constraint that should persist across sessions
|
|
198
|
+
- `qualityMultiplier = 0.7 + 0.6·qualityScore` (0.7× penalty … 1.3× boost).
|
|
199
|
+
- `coActivationBoost = 1 + min(coCount/10, 0.15)` (docs surfaced together get up to +15%).
|
|
200
|
+
- Length normalization penalizes verbose entries (floor 30%); frequency boost capped at +10%.
|
|
201
|
+
- **Pinned docs: +0.3 additive** (capped at 1.0).
|
|
202
|
+
- **`query` tool (v0.13.0+):** non-recency queries use retrieval-tuned **0.70·search + 0.15·recency + 0.15·confidence**. `search`/`vsearch`/`memory_retrieve`/`context-surfacing` keep the 0.50/0.25/0.25 default.
|
|
203
|
+
- **Recency intent** ("latest"/"recent"/"last session") switches all to **0.10·search + 0.70·recency + 0.20·confidence**.
|
|
561
204
|
|
|
562
|
-
**
|
|
205
|
+
**Content-type half-lives:** decision / deductive / preference / hub / antipattern = ∞ (never decay) · project 120d · research 90d · problem / milestone / note 60d · conversation / progress 45d · handoff 30d. Half-lives extend up to 3× for frequently-accessed memories. Attention decay: non-durable types (handoff, progress, conversation, note, project) lose 5% confidence/week without access; decision / deductive / preference / hub / research / antipattern are exempt.
|
|
563
206
|
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
Temporarily hides from context surfacing until a date.
|
|
567
|
-
|
|
568
|
-
**Proactive triggers:**
|
|
569
|
-
- A memory keeps surfacing but isn't relevant to current work
|
|
570
|
-
- User says "not now" / "later" / "ignore this for now"
|
|
571
|
-
- Seasonal or time-boxed content
|
|
572
|
-
|
|
573
|
-
### Forget (`memory_forget`)
|
|
574
|
-
|
|
575
|
-
Permanently deactivates. Use sparingly — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
|
|
576
|
-
|
|
577
|
-
### Contradiction Auto-Resolution
|
|
578
|
-
|
|
579
|
-
When `decision-extractor` detects a new decision contradicting an old one, the old decision's confidence is lowered automatically. No manual intervention needed.
|
|
580
|
-
|
|
581
|
-
**v0.7.1 merge-time contradiction gate:** The consolidation worker adds a second layer at merge time. Before Phase 2 merges a new pattern into an existing consolidated observation, it runs a deterministic heuristic (negation asymmetry, number/date mismatch) followed by an LLM confirmation. When confidence crosses `CLAWMEM_CONTRADICTION_MIN_CONFIDENCE` (default 0.5), the merge is blocked and one of two policies applies via `CLAWMEM_CONTRADICTION_POLICY`:
|
|
582
|
-
|
|
583
|
-
- `link` (default) — insert a new consolidated row and create a `contradicts` edge in `memory_relations`. Both remain queryable.
|
|
584
|
-
- `supersede` — insert the new row and mark the old row `status='inactive'` with `invalidated_at`/`superseded_by` set. The old row is filtered from retrieval but preserved for audit.
|
|
585
|
-
|
|
586
|
-
Phase 3 deductive synthesis applies the same `contradicts` link for any draft that matches a prior deductive observation with conflicting content.
|
|
587
|
-
|
|
588
|
-
---
|
|
589
|
-
|
|
590
|
-
## Anti-Patterns
|
|
591
|
-
|
|
592
|
-
- Do NOT manually pick query/intent_search/search when `memory_retrieve` can auto-route.
|
|
593
|
-
- Do NOT call MCP tools every turn — 3-rule escalation gate is the only trigger.
|
|
594
|
-
- Do NOT re-search what's already in `<vault-context>`.
|
|
595
|
-
- Do NOT run `status` routinely. Only when retrieval feels broken or after large ingestion.
|
|
596
|
-
- Do NOT pin everything — pin is for persistent high-priority items.
|
|
597
|
-
- Do NOT forget memories to "clean up" — let confidence decay and contradiction detection handle it.
|
|
598
|
-
- Do NOT run `build_graphs` after every reindex — A-MEM creates per-doc links automatically.
|
|
599
|
-
- Do NOT run `clawmem mine` autonomously — it is a bulk ingestion command. Suggest it to the user when they mention old conversation exports, but let them run it. **v0.7.2 adds `--synthesize`** — an opt-in post-import LLM fact extraction pass. Also requires user consent because it drives one extra LLM call per conversation doc. Suggest both together when the user wants searchable structured memory from raw chat exports.
|
|
600
|
-
- Do NOT use `diary_write` in Claude Code — hooks capture this automatically. Diary is for non-hooked environments only (Hermes, Gemini, plain MCP).
|
|
601
|
-
- Do NOT use `kg_query` for causal "why" questions — use `intent_search` or `memory_retrieve`. `kg_query` returns structured entity facts (SPO triples), not reasoning chains.
|
|
207
|
+
→ full derivation: [`docs/concepts/composite-scoring.md`](docs/concepts/composite-scoring.md).
|
|
602
208
|
|
|
603
209
|
---
|
|
604
210
|
|
|
605
|
-
##
|
|
606
|
-
|
|
607
|
-
**Active Memory coexistence:** ClawMem is fully compatible with OpenClaw's Active Memory plugin (v2026.4.10+). They search different backends and inject into different prompt regions, both can run simultaneously. The deployment options below control native memory search (`memorySearch.extraPaths`), not Active Memory.
|
|
608
|
-
|
|
609
|
-
**memory-core dreaming sidecar coexistence (v2026.4.18+, #65411):** When ClawMem owns the memory slot AND `plugins.entries.memory-core.config.dreaming.enabled = true`, OpenClaw loads `memory-core`'s dreaming engine alongside ClawMem (rest of `memory-core` stays unloaded). Default after `openclaw plugins enable clawmem` is `dreaming.enabled = false` (ClawMem-only). Set it `true` to keep the dreaming output stream (`memory/dreaming/{phase}/YYYY-MM-DD.md`) alongside ClawMem.
|
|
610
|
-
|
|
611
|
-
**OpenClaw v2026.4.11+ required for ClawMem v0.10.0+.** v2026.4.11 tightened plugin discovery (requires `package.json` with `openclaw.extensions`, rejects symlinked plugin directories). ClawMem v0.10.0 ships the new discovery manifest and defaults `clawmem setup openclaw` to recursive copy (not symlink). v2026.4.10 earlier fixed the #64192 config-normalization bug that dropped the `contextEngine` slot, which is now moot on v0.10.0+ because v0.10.0 uses the `memory` slot instead.
|
|
612
|
-
|
|
613
|
-
**ClawMem v0.10.0 plugin kind:** `memory` (not `context-engine`). Enable with `openclaw plugins enable clawmem`, which also disables `memory-core` / `memory-lancedb` in one step.
|
|
614
|
-
|
|
615
|
-
### Option 1: ClawMem Exclusive (Recommended)
|
|
616
|
-
|
|
617
|
-
ClawMem handles 100% of structured memory. Disable native memory search:
|
|
618
|
-
|
|
619
|
-
```bash
|
|
620
|
-
openclaw config set agents.defaults.memorySearch.extraPaths "[]"
|
|
621
|
-
```
|
|
622
|
-
|
|
623
|
-
**Distribution:** Hooks 90%, MCP tools 10%.
|
|
624
|
-
|
|
625
|
-
### Option 2: Hybrid
|
|
626
|
-
|
|
627
|
-
Run both ClawMem and OpenClaw native memory search.
|
|
628
|
-
|
|
629
|
-
```bash
|
|
630
|
-
openclaw config set agents.defaults.memorySearch.extraPaths '["~/documents", "~/notes"]'
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
**Tradeoffs:** Redundant recall but 10-15% context window waste from duplicate facts.
|
|
634
|
-
|
|
635
|
-
### Multi-user install gotcha (v2026.4.11+)
|
|
636
|
-
|
|
637
|
-
If the gateway runs as a dedicated system user (e.g. `openclaw`) different from the user who runs `clawmem setup openclaw` (e.g. `sciros`), the copied plugin dir is rejected with `suspicious ownership (uid=X, expected uid=Y or root)`. Fix: `sudo chown -R <gateway-user>:<gateway-group> ~/.openclaw/extensions/clawmem`. Single-user installs are not affected.
|
|
638
|
-
|
|
639
|
-
Also: if the gateway user cannot traverse `~/<installer>/.openclaw/` (directory mode 700), the gateway fails to start with `Missing config. Run openclaw setup or set gateway.mode=local`. Fix: `sudo chmod 750 ~/<installer>/.openclaw` and ensure the gateway user is in the owning group.
|
|
211
|
+
## Memory lifecycle (pin / snooze / forget — manual tools)
|
|
640
212
|
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
213
|
+
- **`memory_pin`** (+0.3 boost, persistent surfacing) — PROACTIVELY when: user says "remember this"/"important"; an architecture/critical decision was just made; a user preference/constraint should persist across sessions. Do NOT pin routine/session-specific items.
|
|
214
|
+
- **`memory_snooze`** — PROACTIVELY when a memory keeps surfacing but isn't relevant now, user says "not now"/"later", or content is time-boxed.
|
|
215
|
+
- **`memory_forget`** — only when genuinely wrong or permanently obsolete. Prefer snooze for temporary suppression.
|
|
216
|
+
- **Contradiction auto-resolution:** when `decision-extractor` detects a new decision contradicting an old one, the old one's confidence is lowered automatically — no manual action needed.
|
|
644
217
|
|
|
645
218
|
---
|
|
646
219
|
|
|
647
|
-
##
|
|
648
|
-
|
|
649
|
-
### Install
|
|
650
|
-
|
|
651
|
-
Copy or symlink `src/hermes/` into `hermes-agent/plugins/memory/clawmem/`. Set `memory.provider: clawmem` in Hermes config.
|
|
652
|
-
|
|
653
|
-
### How it works
|
|
654
|
-
|
|
655
|
-
Plugin implements Hermes's `MemoryProvider` ABC:
|
|
656
|
-
- `prefetch()` — context-surfacing hook (automatic per turn)
|
|
657
|
-
- `on_session_end()` — extraction hooks in parallel (decision-extractor, handoff-generator, feedback-loop)
|
|
658
|
-
- `on_pre_compress()` — precompact-extract (side effect only)
|
|
659
|
-
- 5 agent tools via REST: `clawmem_retrieve`, `clawmem_get`, `clawmem_session_log`, `clawmem_timeline`, `clawmem_similar`
|
|
220
|
+
## Operational gotchas (agent-facing)
|
|
660
221
|
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
`clawmem` binary on PATH + `clawmem serve` running (external) or `CLAWMEM_SERVE_MODE=managed`. Python 3.10+.
|
|
222
|
+
- **Empty `context-surfacing`** → prompt < 20 chars, starts with `/`, or nothing scored above threshold. Check `clawmem status` (doc counts) + embedding coverage.
|
|
223
|
+
- **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed` or wait for the embed timer.
|
|
224
|
+
- **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs` (temporal backbone + semantic edges). Otherwise don't run it after every reindex — A-MEM links per-doc automatically.
|
|
225
|
+
- **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert, non-discriminating scores, silently collapsing ranking to RRF. The reranker is a served sidecar, not a bundled model — verify it discriminates, don't assume liveness = correctness.
|
|
226
|
+
- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → almost always the context-surfacing hook's **cold-start**, NOT inference: a fresh Bun process + opening a large `index.sqlite` + a cold OS page cache. Warm calls are sub-second. On a memory-constrained host (e.g. WSL with a low memory cap) or a large vault, the cache is evicted between turns so it **recurs on certain turns**. A timed-out hook silently drops that turn's `<vault-context>` (degraded recall, no error). **Durable fix: give the host enough RAM to keep the index + Bun modules cached**; raising the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) is only a secondary margin — avoid 15s+ as a standing default since the hook blocks prompt submission. Full detail: [`docs/troubleshooting.md`](docs/troubleshooting.md) → *Hooks slow or near timeout* / *Tuning the context-surfacing hook timeout*.
|
|
227
|
+
- **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [`docs/troubleshooting.md`](docs/troubleshooting.md). This skill does not duplicate it.
|
|
668
228
|
|
|
669
229
|
---
|
|
670
230
|
|
|
671
|
-
##
|
|
231
|
+
## Anti-patterns
|
|
672
232
|
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
Symptom: Vector search returns no results but BM25 works
|
|
683
|
-
-> Missing embeddings. Watcher indexes but does NOT embed.
|
|
684
|
-
-> Fix: Run `clawmem embed` or wait for daily embed timer.
|
|
685
|
-
|
|
686
|
-
Symptom: context-surfacing hook returns empty
|
|
687
|
-
-> Prompt too short (<20 chars), starts with `/`, or no docs above threshold.
|
|
688
|
-
-> Fix: Check `clawmem status` for doc counts. Check `clawmem embed` for embedding coverage.
|
|
689
|
-
|
|
690
|
-
Symptom: intent_search returns weak results for WHY/ENTITY
|
|
691
|
-
-> Graph may be sparse (few A-MEM edges).
|
|
692
|
-
-> Fix: Run `build_graphs` to add temporal backbone + semantic edges.
|
|
693
|
-
|
|
694
|
-
Symptom: Watcher fires but collections show 0 docs
|
|
695
|
-
-> Bun.Glob does not support brace expansion {a,b,c}.
|
|
696
|
-
-> Fixed: indexer.ts splits brace patterns into individual Glob scans.
|
|
697
|
-
|
|
698
|
-
Symptom: Watcher fires but wrong collection processes events
|
|
699
|
-
-> Collection prefix matching returns first match. Parent paths match before children.
|
|
700
|
-
-> Fixed: cmdWatch() sorts by path length descending (most specific first).
|
|
701
|
-
|
|
702
|
-
Symptom: reindex --force crashes with UNIQUE constraint
|
|
703
|
-
-> Force deactivates rows but UNIQUE(collection, path) doesn't discriminate by active flag.
|
|
704
|
-
-> Fixed: indexer.ts reactivates inactive rows instead of inserting.
|
|
705
|
-
|
|
706
|
-
Symptom: `clawmem update` crashes with "Binding expected string, TypedArray, boolean, number, bigint or null"
|
|
707
|
-
-> YAML frontmatter values like `title: 2023-09-27` or `title: true` are coerced by gray-matter
|
|
708
|
-
into Date objects or booleans. Bun's SQLite driver rejects these as bind parameters.
|
|
709
|
-
-> Fixed v0.4.2: `parseDocument()` runtime-checks all frontmatter fields via `str()` helper.
|
|
710
|
-
-> Affects: title, domain, workstream, content_type, review_by.
|
|
711
|
-
|
|
712
|
-
Symptom: CLI reindex/update falls back to node-llama-cpp
|
|
713
|
-
-> GPU env vars only in systemd drop-in, not in wrapper script.
|
|
714
|
-
-> Fixed: bin/clawmem wrapper exports CLAWMEM_EMBED_URL/LLM_URL/RERANK_URL defaults.
|
|
715
|
-
|
|
716
|
-
Symptom: "UserPromptSubmit hook error" on context-surfacing hook (intermittent)
|
|
717
|
-
-> SQLite contention between watcher and hook. Watcher processes filesystem events and holds
|
|
718
|
-
brief write locks. If the hook fires during a lock, it can exceed its timeout. More likely
|
|
719
|
-
during active conversations with frequent file changes.
|
|
720
|
-
-> v0.1.6 fix: watcher no longer processes session transcript .jsonl files (only .beads/*.jsonl),
|
|
721
|
-
eliminating the most common source of contention.
|
|
722
|
-
-> Default hook timeout is 8s (since v0.1.1). If you have an older install, re-run
|
|
723
|
-
`clawmem setup hooks`. If persistent, restart the watcher: `systemctl --user restart
|
|
724
|
-
clawmem-watcher.service`. Healthy memory is under 100MB — if 400MB+, restart clears it.
|
|
725
|
-
-> v0.2.4 fix: hook's SQLite busy_timeout was 500ms — too tight. During A-MEM enrichment
|
|
726
|
-
or heavy indexing, watcher write locks exceed 500ms, causing SQLITE_BUSY. Raised to
|
|
727
|
-
5000ms (matches MCP server). Still completes within the 8s outer timeout.
|
|
728
|
-
|
|
729
|
-
Symptom: WSL hangs or becomes unresponsive during long sessions / watcher has 100K+ FDs
|
|
730
|
-
-> Pre-v0.2.3: fs.watch(recursive: true) registered inotify watches on EVERY subdirectory,
|
|
731
|
-
including excluded dirs (gits/, node_modules/, .git/). Broad collection paths like
|
|
732
|
-
~/Projects with 67K subdirs exhausted inotify limits.
|
|
733
|
-
-> v0.2.3 fix: watcher walks dir trees at startup, skips excluded subtrees, watches
|
|
734
|
-
non-excluded dirs individually. 500-dir cap per collection path.
|
|
735
|
-
-> Diagnosis: `ls /proc/$(pgrep -f "clawmem.*watch")/fd | wc -l` — healthy < 15K.
|
|
736
|
-
-> If still high: narrow broad collection paths. See docs/troubleshooting.md.
|
|
737
|
-
```
|
|
233
|
+
- ❌ Manually pick `query`/`intent_search`/`search` when `memory_retrieve` can auto-route → ✅ `memory_retrieve` first.
|
|
234
|
+
- ❌ Call MCP tools every turn → ✅ only when the 3-rule gate fires.
|
|
235
|
+
- ❌ Re-search what's already in `<vault-context>`.
|
|
236
|
+
- ❌ Run `status` routinely → ✅ only when retrieval feels broken or after large ingestion.
|
|
237
|
+
- ❌ Pin everything → ✅ pin only persistent high-priority items.
|
|
238
|
+
- ❌ Forget memories to "clean up" → ✅ let decay + contradiction detection handle it.
|
|
239
|
+
- ❌ `build_graphs` after every reindex → ✅ only after bulk ingestion or when graph traversal is weak.
|
|
240
|
+
- ❌ `diary_write` in Claude Code → ✅ hooks capture this automatically (diary is for non-hooked envs only).
|
|
241
|
+
- ❌ `kg_query` for causal "why" → ✅ `intent_search` (kg_query is entity facts, not reasoning chains).
|
|
738
242
|
|
|
739
243
|
---
|
|
740
244
|
|
|
741
|
-
##
|
|
245
|
+
## Curator agent
|
|
742
246
|
|
|
743
|
-
|
|
247
|
+
Maintenance agent for Tier-3 work the main agent neglects. Invoke: **"curate memory" / "run curator" / "memory maintenance"**. Six phases: (1) health snapshot, (2) lifecycle triage (pin/snooze/propose-forget — never auto-confirms), (3) retrieval health probes, (4) reflect + consolidate `--dry-run`, (5) conditional graph rebuild, (6) collection hygiene. Safety rails: never auto-confirms forget, never runs embed, never edits config.
|
|
744
248
|
|
|
745
|
-
|
|
249
|
+
## Tool selection (one-liner)
|
|
746
250
|
|
|
747
|
-
```bash
|
|
748
|
-
# IO6a: per-prompt context injection (pipe prompt on stdin)
|
|
749
|
-
echo "user query" | clawmem surface --context --stdin
|
|
750
|
-
|
|
751
|
-
# IO6b: per-session bootstrap injection (pipe session ID on stdin)
|
|
752
|
-
echo "session-id" | clawmem surface --bootstrap --stdin
|
|
753
|
-
```
|
|
754
|
-
|
|
755
|
-
### Enrichment Commands
|
|
756
|
-
|
|
757
|
-
```bash
|
|
758
|
-
clawmem reindex --enrich # Full A-MEM pipeline on ALL docs (entity extraction,
|
|
759
|
-
# link generation, memory evolution). Use after major upgrades.
|
|
760
|
-
# Without --enrich, reindex only refreshes metadata for changed docs.
|
|
761
|
-
```
|
|
762
|
-
|
|
763
|
-
### Browse and Analysis
|
|
764
|
-
|
|
765
|
-
```bash
|
|
766
|
-
clawmem list [-n N] [-c col] # Browse recent documents (--json for machine output)
|
|
767
|
-
clawmem reflect [N] # Cross-session reflection (last N days, default 14)
|
|
768
|
-
# Recurring themes, antipatterns, co-activation clusters
|
|
769
|
-
clawmem consolidate [--dry-run] # Find and archive duplicate low-confidence documents
|
|
770
|
-
# Jaccard similarity within same collection
|
|
771
251
|
```
|
|
772
|
-
|
|
773
|
-
### Session Focus Topic (v0.9.0 §11.4)
|
|
774
|
-
|
|
775
|
-
Per-session topic biasing for context-surfacing. Writes a focus file at `~/.cache/clawmem/sessions/<session_id>.focus` that steers query expansion, reranking, snippet extraction, and post-composite-score topic boost (1.4x match / 0.75x demote, NO-OP on zero matches). Session-isolated — never writes to SQLite or lifecycle columns. Session ID resolved from `--session-id <id>` > `CLAUDE_SESSION_ID` env > `CLAWMEM_SESSION_ID` env.
|
|
776
|
-
|
|
777
|
-
**When to use:** user says "focus on X for this session" / "only surface Y right now" / "let's work on Z." Clear at end of subsession to return to baseline.
|
|
778
|
-
|
|
779
|
-
```bash
|
|
780
|
-
clawmem focus set "authentication flow" # uses CLAUDE_SESSION_ID env
|
|
781
|
-
clawmem focus set "authentication flow" --session-id abc123 # explicit session id
|
|
782
|
-
clawmem focus show --session-id abc123
|
|
783
|
-
clawmem focus clear --session-id abc123
|
|
252
|
+
memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) -> multi_get -> search/vsearch (spot checks)
|
|
784
253
|
```
|
|
785
254
|
|
|
786
|
-
|
|
787
255
|
---
|
|
788
256
|
|
|
789
|
-
##
|
|
790
|
-
|
|
791
|
-
- QMD retrieval (BM25, vector, RRF, rerank, query expansion) is forked into ClawMem. Do not call standalone QMD tools.
|
|
792
|
-
- SAME (composite scoring), MAGMA (intent + graph), A-MEM (self-evolving notes) layer on top of QMD substrate.
|
|
793
|
-
- Three inference services on local or remote GPU — `llama-server` for all three in the default stack; the SOTA stack serves the reranker as a transformers sidecar. Wrapper defaults to `localhost:8088/8089/8090`.
|
|
794
|
-
- `CLAWMEM_NO_LOCAL_MODELS=false` (default) allows in-process fallback. Set `true` for remote-only to fail fast.
|
|
795
|
-
- Consolidation worker (`CLAWMEM_ENABLE_CONSOLIDATION=true`) backfills unenriched docs and runs Phase 2 merge / Phase 3 deductive synthesis. **v0.8.2:** hosted by either `clawmem watch` (long-lived, canonical) or `clawmem mcp` (per-session fallback); every tick acquires a `light-consolidation` `worker_leases` row before doing work, so dual-hosting against the same vault is safe.
|
|
796
|
-
- Beads integration: `syncBeadsIssues()` queries `bd` CLI (Dolt backend, v0.58.0+), creates markdown docs, maps dependency edges into `memory_relations`. Watcher auto-triggers on `.beads/` changes; `beads_sync` MCP for manual sync.
|
|
797
|
-
- HTTP REST API: `clawmem serve [--port 7438]` — optional REST server on localhost. Search, retrieval, lifecycle, and graph traversal. `POST /retrieve` mirrors `memory_retrieve` with auto-routing (keyword/semantic/causal/timeline/hybrid). `POST /search` provides direct mode selection. Bearer token auth via `CLAWMEM_API_TOKEN` env var (disabled if unset).
|
|
798
|
-
- OpenClaw memory plugin (v0.10.0+): `clawmem setup openclaw` — registers as native OpenClaw memory plugin (`kind: memory`). Dual-mode: shares vault with Claude Code hooks. Hook wiring on the plugin-hook bus: `before_prompt_build` is the **load-bearing** path — it runs prompt-aware retrieval AND the pre-emptive `precompact-extract` synchronously when token usage approaches the compaction threshold, so state is captured before the LLM call that could trigger compaction. `agent_end` runs decision-extractor + handoff-generator + feedback-loop in parallel (fire-and-forget at OpenClaw's call site, plus a 30s default void-hook timeout from OpenClaw v2026.4.26+ that logs slow handlers but does not cancel the underlying postrun work). `before_compaction` is **defense-in-depth fallback only** — fire-and-forget, races the compactor, exists for the rare case where the proximity heuristic in `before_prompt_build` missed a sudden token jump. `session_start` registers the session + caches first-turn bootstrap context. The §14.3 migration removed the `ClawMemContextEngine` class and moved the plugin from the `context-engine` slot to the `memory` slot. Requires OpenClaw v2026.4.11+ (earlier versions do not support the new discovery contract).
|
|
799
|
-
- Hermes Agent MemoryProvider plugin: `src/hermes/` — Python plugin for Hermes's memory system. Shell-out hooks for lifecycle (prefetch, extraction, precompact), REST API for tools. Plugin-managed transcript JSONL bridges Hermes turn pairs to ClawMem file format. Shares vault with Claude Code and OpenClaw. **Preferred install path:** `$HERMES_HOME/plugins/clawmem/` (Hermes #10529 user-plugin discovery, v2026.4.13+) — survives `git pull` of hermes-agent. The bundled `hermes-agent/plugins/memory/clawmem/` path still works. **Agent-context isolation:** read-side hooks always run; write-side surfaces (`sync_turn`, `on_session_end`, `on_pre_compress`) early-return when `agent_context != "primary"` so cron/subagent state never reaches the vault.
|
|
800
|
-
|
|
801
|
-
## Tool Selection (one-liner)
|
|
802
|
-
|
|
803
|
-
```
|
|
804
|
-
ClawMem escalation: memory_retrieve(query) | query(compact=true) | intent_search(why/when/entity) | query_plan(multi-topic) -> multi_get -> search/vsearch (spot checks)
|
|
805
|
-
```
|
|
806
|
-
|
|
807
|
-
## Curator Agent
|
|
808
|
-
|
|
809
|
-
Maintenance agent for Tier 3 operations the main agent typically neglects. Install with `clawmem setup curator`.
|
|
810
|
-
|
|
811
|
-
**Invoke:** "curate memory", "run curator", or "memory maintenance"
|
|
257
|
+
## Setup / config / internals → AGENTS.md + docs/
|
|
812
258
|
|
|
813
|
-
**
|
|
814
|
-
1. Health snapshot — status, index_stats, lifecycle_status, doctor
|
|
815
|
-
2. Lifecycle triage — pin high-value unpinned memories, snooze stale content, propose forget candidates (never auto-confirms)
|
|
816
|
-
3. Retrieval health check — 5 probes (BM25, vector, hybrid, intent/graph, lifecycle)
|
|
817
|
-
4. Maintenance — reflect (cross-session patterns), consolidate --dry-run (dedup candidates)
|
|
818
|
-
5. Graph rebuild — conditional on probe results and embedding state
|
|
819
|
-
6. Collection hygiene — orphan detection, content type distribution
|
|
259
|
+
This skill is **operations-only**. For installation, inference-server setup (the embedding/LLM/reranker services — the SOTA reranker is a **seq-cls sidecar, not a GGUF**), environment variables, systemd units, indexing/collection config, graph internals, and the OpenClaw (`kind: memory`) / Hermes (`MemoryProvider`) plugins, see [`AGENTS.md`](AGENTS.md) and [`docs/`](docs/):
|
|
820
260
|
|
|
821
|
-
|
|
261
|
+
- Inference stack choice + server setup → [`docs/guides/inference-services.md`](docs/guides/inference-services.md)
|
|
262
|
+
- All environment variables → [`docs/reference/configuration.md`](docs/reference/configuration.md)
|
|
263
|
+
- Cloud embedding → [`docs/guides/cloud-embedding.md`](docs/guides/cloud-embedding.md)
|
|
264
|
+
- Setup (hooks / mcp / systemd) → [`docs/guides/setup-hooks.md`](docs/guides/setup-hooks.md), [`docs/guides/setup-mcp.md`](docs/guides/setup-mcp.md), [`docs/guides/systemd-services.md`](docs/guides/systemd-services.md)
|
|
265
|
+
- Internals (pipelines, graph, entities) → [`docs/internals/`](docs/internals/)
|
|
266
|
+
- OpenClaw / Hermes plugins → [`docs/guides/openclaw-plugin.md`](docs/guides/openclaw-plugin.md), [`docs/guides/hermes-plugin.md`](docs/guides/hermes-plugin.md)
|
|
267
|
+
- Troubleshooting → [`docs/troubleshooting.md`](docs/troubleshooting.md)
|