clawmem 0.1.1 → 0.1.3

package/README.md

@@ -8,7 +8,7 @@
 
 ClawMem fuses recent research into a retrieval-augmented memory layer that agents actually use. The hybrid architecture combines [QMD](https://github.com/tobi/qmd)-derived multi-signal retrieval (BM25 + vector search + reciprocal rank fusion + query expansion + cross-encoder reranking), [SAME](https://github.com/sgx-labs/statelessagent)-inspired composite scoring (recency decay, confidence, content-type half-lives, co-activation reinforcement), [MAGMA](https://arxiv.org/abs/2501.13956)-style intent classification with multi-graph traversal (semantic, temporal, and causal beam search), and [A-MEM](https://arxiv.org/abs/2510.02178) self-evolving memory notes that enrich documents with keywords, tags, and causal links between entries. Pattern extraction from [Engram](https://github.com/Gentleman-Programming/engram) adds deduplication windows, frequency-based durability scoring, and temporal navigation.
 
-Two integration paths: Claude Code hooks paired with an MCP server, or a native OpenClaw ContextEngine plugin. Both write to the same local SQLite vault. A decision captured during a Claude Code session shows up immediately when an OpenClaw agent picks up the same project.
+Integrates via Claude Code hooks, an MCP server (works with any MCP-compatible client including OpenClaw), or a native OpenClaw ContextEngine plugin. All paths write to the same local SQLite vault. A decision captured during a Claude Code session shows up immediately when an OpenClaw agent picks up the same project.
 
 TypeScript on Bun. MIT License.
 
@@ -61,8 +61,21 @@ Runs fully local with no API keys and no cloud services. Integrates via Claude C
 
 ### Prerequisites
 
-- [Bun](https://bun.sh) v1.0+
-- SQLite with FTS5 support (included with Bun)
+**Required:**
+
+- [Bun](https://bun.sh) v1.0+ — runtime for ClawMem. On Linux, install via `curl -fsSL https://bun.sh/install | bash` (not snap — snap Bun cannot read stdin, which breaks hooks).
+- SQLite with FTS5 — included with Bun. On macOS, install `brew install sqlite` for extension loading support (ClawMem detects and uses Homebrew SQLite automatically).
+
+**Optional (for better performance):**
+
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) (`llama-server`) — for dedicated GPU inference. Without it, `node-llama-cpp` runs models in-process (auto-downloads on first use). GPU servers give better throughput and prevent silent CPU fallback.
+- systemd (Linux) or launchd (macOS) — for persistent background services (watcher, embed timer, GPU servers). ClawMem ships systemd unit templates; macOS users can create equivalent launchd plists. See [systemd services](docs/guides/systemd-services.md).
+
+**Optional integrations:**
+
+- [Claude Code](https://docs.anthropic.com/en/docs/claude-code) — for hooks + MCP integration
+- [OpenClaw](https://github.com/openclawai/openclaw) — for ContextEngine plugin integration
+- [bd CLI](https://github.com/dolthub/dolt) v0.58.0+ — for Beads issue tracker sync (only if using Beads)
 
 ### Install from npm (recommended)
 
@@ -100,6 +113,8 @@ After installing, here's the full journey from zero to working memory:
 
 **Fastest path:** Step 1 alone gets you a working system with in-process CPU/GPU inference and default models — no manual model downloads or service configuration needed. Steps 2-4 are optional upgrades for better performance. Steps 5-6 are where you customize what gets indexed and how your agent connects.
 
+**Customize what gets indexed:** Each collection has a `pattern` field in `~/.config/clawmem/config.yaml` (default: `**/*.md`). Tailor it per collection — index project docs, research notes, decision records, Obsidian vaults, or anything else your agents should know about. The more relevant content in the vault, the better retrieval works. See the [quickstart](docs/quickstart.md#customize-index-patterns) for config examples.
+
 ### Quick start commands
 
 ```bash
@@ -130,7 +145,7 @@ ClawMem integrates via hooks (`settings.json`) and an MCP stdio server. Hooks ha
 
 ```bash
 clawmem setup hooks    # Install lifecycle hooks (SessionStart, UserPromptSubmit, Stop, PreCompact)
-clawmem setup mcp      # Register MCP server in ~/.claude.json (20+ agent tools)
+clawmem setup mcp      # Register MCP server in ~/.claude.json (28 tools)
 ```
 
 **Automatic (90%):** `context-surfacing` injects relevant memory on every prompt. `postcompact-inject` re-injects state after compaction. `decision-extractor`, `handoff-generator`, `feedback-loop` capture session state on stop.
@@ -157,7 +172,7 @@ Disable OpenClaw's native memory and `memory-lancedb` auto-recall/capture to avo
 openclaw config set agents.defaults.memorySearch.extraPaths "[]"
 ```
 
-**Alternative:** You can also use the Claude Code-style hooks + MCP approach with OpenClaw (`clawmem setup hooks && clawmem setup mcp`). This works but bypasses OpenClaw's ContextEngine lifecycle - you lose token budget awareness, native compaction orchestration, and the `afterTurn()` message pipeline. The ContextEngine plugin is recommended for new OpenClaw setups.
+**Alternative:** OpenClaw agents can also use ClawMem's MCP server directly (`clawmem setup mcp`), with or without hooks. This gives full access to all 28 MCP tools but bypasses OpenClaw's ContextEngine lifecycle, so you lose token budget awareness, native compaction orchestration, and the `afterTurn()` message pipeline. The ContextEngine plugin is recommended for new OpenClaw setups; MCP is available as an additional or standalone integration.
 
 #### Dual-Mode Operation
 
@@ -388,7 +403,7 @@ llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf \
 
 ### MCP Server
 
-ClawMem exposes 26 MCP tools via the [Model Context Protocol](https://modelcontextprotocol.io) and an optional HTTP REST API. Any MCP-compatible client or HTTP client can use it.
+ClawMem exposes 28 MCP tools via the [Model Context Protocol](https://modelcontextprotocol.io) and an optional HTTP REST API. Any MCP-compatible client or HTTP client can use it.
 
 **Claude Code (automatic):**
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clawmem",
-  "version": "0.1.1",
+  "version": "0.1.3",
   "description": "On-device context engine and memory for AI agents. Claude Code and OpenClaw. Hooks + MCP server + hybrid RAG search.",
   "type": "module",
   "bin": {

package/src/config.ts

@@ -71,12 +71,16 @@ export interface ProfileConfig {
   useVector: boolean;
   vectorTimeout: number;
   minScore: number;
+  /** Budget-aware escalation: if fast path finishes early, spend remaining time on expansion + reranking */
+  deepEscalation: boolean;
+  /** Max time (ms) allowed for the fast path before escalation is considered */
+  escalationBudgetMs: number;
 }
 
 export const PROFILES: Record<PerformanceProfile, ProfileConfig> = {
-  speed:    { tokenBudget: 400,  maxResults: 5,  useVector: false, vectorTimeout: 0,    minScore: 0.55 },
-  balanced: { tokenBudget: 800,  maxResults: 10, useVector: true,  vectorTimeout: 900,  minScore: 0.45 },
-  deep:     { tokenBudget: 1200, maxResults: 15, useVector: true,  vectorTimeout: 2000, minScore: 0.35 },
+  speed:    { tokenBudget: 400,  maxResults: 5,  useVector: false, vectorTimeout: 0,    minScore: 0.55, deepEscalation: false, escalationBudgetMs: 0 },
+  balanced: { tokenBudget: 800,  maxResults: 10, useVector: true,  vectorTimeout: 900,  minScore: 0.45, deepEscalation: false, escalationBudgetMs: 0 },
+  deep:     { tokenBudget: 1200, maxResults: 15, useVector: true,  vectorTimeout: 2000, minScore: 0.25, deepEscalation: true,  escalationBudgetMs: 4000 },
 };
 
 export function getActiveProfile(): ProfileConfig {

package/src/hooks/context-surfacing.ts

@@ -7,7 +7,8 @@
  */
 
 import type { Store, SearchResult } from "../store.ts";
-import { DEFAULT_EMBED_MODEL, extractSnippet } from "../store.ts";
+import { DEFAULT_EMBED_MODEL, DEFAULT_QUERY_MODEL, DEFAULT_RERANK_MODEL, extractSnippet, resolveStore } from "../store.ts";
+import { getVaultPath, getActiveProfile } from "../config.ts";
 import type { HookInput, HookOutput } from "../hooks.ts";
 import {
   makeContextOutput,
@@ -29,13 +30,12 @@ import { enrichResults } from "../search-utils.ts";
 import { sanitizeSnippet } from "../promptguard.ts";
 import { shouldSkipRetrieval, isRetrievedNoise } from "../retrieval-gate.ts";
 import { MAX_QUERY_LENGTH } from "../limits.ts";
-import { getActiveProfile } from "../config.ts";
 
 // =============================================================================
 // Config
 // =============================================================================
 
-// Profile-driven defaults (overridden by CLAWMEM_PROFILE env var)
+// Profile-driven defaults (overridden by CLAWMEM_PROFILE env var via E14)
 const DEFAULT_TOKEN_BUDGET = 800;
 const DEFAULT_MAX_RESULTS = 10;
 const DEFAULT_MIN_SCORE = 0.45;
@@ -52,7 +52,7 @@ function getTierConfig(score: number): { snippetLen: number; showMeta: boolean;
 // Directories to never surface
 const FILTERED_PATHS = ["_PRIVATE/", "experiments/", "_clawmem/"];
 
-// File path patterns to extract from prompts (E13: file-aware UserPromptSubmit)
+// File path patterns to extract from prompts (E13 replacement: file-aware UserPromptSubmit)
 const FILE_PATH_RE = /(?:^|\s)((?:\/[\w.@-]+)+(?:\.\w+)?|[\w.@-]+\.(?:ts|js|py|md|sh|yaml|yml|json|toml|rs|go|tsx|jsx|css|html))\b/g;
 
 // =============================================================================
@@ -81,10 +81,11 @@ export async function contextSurfacing(
     return makeEmptyOutput("context-surfacing");
   }
 
-  // Load active performance profile
+  // Load active performance profile (E14)
   const profile = getActiveProfile();
   const maxResults = profile.maxResults;
   const tokenBudget = profile.tokenBudget;
+  const startTime = Date.now();
 
   const isRecency = hasRecencyIntent(prompt);
   const minScore = isRecency ? MIN_COMPOSITE_SCORE_RECENCY : profile.minScore;
@@ -118,7 +119,22 @@ export async function contextSurfacing(
     }
   }
 
-  // File-aware supplemental search (E13): extract file paths/names from prompt
+  // Dual-query: also search skill vault if configured (secondary source)
+  if (getVaultPath("skill")) {
+    try {
+      const skillStore = resolveStore("skill");
+      const skillResults = skillStore.searchFTS(prompt, 5);
+      // Tag skill vault results for identification in output
+      for (const r of skillResults) {
+        (r as any)._fromVault = "skill";
+      }
+      results = [...results, ...skillResults];
+    } catch {
+      // Skill vault unavailable — continue with general results only
+    }
+  }
+
+  // File-aware supplemental search (E13 replacement): extract file paths/names from prompt
   // and run targeted FTS queries to surface file-specific vault context
   const fileMatches = [...prompt.matchAll(FILE_PATH_RE)].map(m => m[1]!.trim()).filter(Boolean);
   if (fileMatches.length > 0) {
@@ -138,6 +154,54 @@ export async function contextSurfacing(
 
   if (results.length === 0) return makeEmptyOutput("context-surfacing");
 
+  // Budget-aware deep escalation (deep profile only):
+  // If the fast path finished quickly and found results, spend remaining time budget
+  // on query expansion (discovers new candidates) and cross-encoder reranking (reorders).
+  if (profile.deepEscalation && results.length >= 2) {
+    const elapsed = Date.now() - startTime;
+    if (elapsed < profile.escalationBudgetMs) {
+      try {
+        // Phase 1: Query expansion — discover candidates BM25+vector missed
+        const expanded = await store.expandQuery(prompt, DEFAULT_QUERY_MODEL);
+        if (expanded.length > 0) {
+          const seen = new Set(results.map(r => r.filepath));
+          for (const eq of expanded.slice(0, 3)) {
+            if (Date.now() - startTime > 6000) break; // hard stop at 6s
+            const ftsExp = store.searchFTS(eq, 5);
+            for (const r of ftsExp) {
+              if (!seen.has(r.filepath)) {
+                seen.add(r.filepath);
+                results.push(r);
+              }
+            }
+          }
+        }
+
+        // Phase 2: Cross-encoder reranking — reorder with deeper relevance signal
+        if (Date.now() - startTime < 6000 && results.length >= 3) {
+          const toRerank = results.slice(0, 15).map(r => ({
+            file: r.filepath,
+            text: (r.body || "").slice(0, 2000),
+          }));
+          const reranked = await store.rerank(prompt, toRerank, DEFAULT_RERANK_MODEL);
+          if (reranked.length > 0) {
+            const rerankedMap = new Map(reranked.map(r => [r.file, r.score]));
+            // Blend: 60% original score + 40% reranker score for stability
+            for (const r of results) {
+              const rerankScore = rerankedMap.get(r.filepath);
+              if (rerankScore !== undefined) {
+                r.score = 0.6 * r.score + 0.4 * rerankScore;
+              }
+            }
+            results.sort((a, b) => b.score - a.score);
+          }
+        }
+      } catch {
+        // Escalation failed (GPU down, timeout, etc.) — continue with fast-path results
+      }
+    }
+  }
+
   // Filter out private/excluded paths
   results = results.filter(r =>
     !FILTERED_PATHS.some(p => r.displayPath.includes(p))
@@ -148,8 +212,12 @@ export async function contextSurfacing(
   // Filter out snoozed documents
   const now = new Date();
   results = results.filter(r => {
+    // filepath is a virtual path (clawmem://collection/path) but findActiveDocument
+    // expects the collection-relative path, not the full virtual path
     const parsed = r.filepath.startsWith('clawmem://') ? r.filepath.replace(/^clawmem:\/\/[^/]+\/?/, '') : r.filepath;
-    const doc = store.findActiveDocument(r.collectionName, parsed);
+    // Use the correct store for skill-vault results
+    const targetStore = (r as any)._fromVault === "skill" ? (() => { try { return resolveStore("skill"); } catch { return store; } })() : store;
+    const doc = targetStore.findActiveDocument(r.collectionName, parsed);
     if (!doc) return true;
     if (doc.snoozed_until && new Date(doc.snoozed_until) > now) return false;
     return true;
@@ -170,8 +238,19 @@ export async function contextSurfacing(
   // Filter out noise results (agent denials, too-short snippets) before enrichment
   results = results.filter(r => !r.body || !isRetrievedNoise(r.body));
 
-  // Enrich with SAME metadata
-  const enriched = enrichResults(store, results, prompt);
+  // Enrich with SAME metadata — route skill-vault results through their own store
+  const generalResults = results.filter(r => !(r as any)._fromVault);
+  const skillResults = results.filter(r => (r as any)._fromVault === "skill");
+  let enriched = enrichResults(store, generalResults, prompt);
+  if (skillResults.length > 0) {
+    try {
+      const skillStore = resolveStore("skill");
+      enriched = [...enriched, ...enrichResults(skillStore, skillResults, prompt)];
+    } catch {
+      // Skill store unavailable — enrich with general store as fallback
+      enriched = [...enriched, ...enrichResults(store, skillResults, prompt)];
+    }
+  }
 
   // Apply composite scoring
   const scored = applyCompositeScoring(enriched, prompt)
@@ -191,6 +270,7 @@ export async function contextSurfacing(
         for (const ca of coActs) {
           const existing = scored.find(r => r.displayPath === ca.path);
           if (existing && existing.compositeScore <= 0.8) {
+            // Boost by 0.1 per co-activation count, capped at +0.2
             existing.compositeScore += Math.min(0.2, 0.1 * Math.min(ca.count, 2));
           }
         }
@@ -202,12 +282,14 @@ export async function contextSurfacing(
   }
 
   // Memory type diversification (E10): ensure procedural results aren't crowded out
+  // If top results are all semantic, promote the best procedural result
   if (scored.length > 3) {
     const top3Types = scored.slice(0, 3).map(r => inferMemoryType(r.displayPath, r.contentType, r.body));
     const hasProc = top3Types.includes("procedural");
     if (!hasProc) {
       const procIdx = scored.findIndex(r => inferMemoryType(r.displayPath, r.contentType, r.body) === "procedural");
       if (procIdx > 3) {
+        // Move the best procedural result to position 3
         const [proc] = scored.splice(procIdx, 1);
         scored.splice(3, 0, proc!);
       }
@@ -225,6 +307,7 @@ export async function contextSurfacing(
   }
 
   // Routing hint: detect query intent signals and prepend a tool routing directive
+  // This makes routing instructions salient at the moment of tool selection (per research)
   const routingHint = detectRoutingHint(prompt);
 
   return makeContextOutput(
@@ -247,14 +330,17 @@ export async function contextSurfacing(
 function detectRoutingHint(prompt: string): string | null {
   const q = prompt.toLowerCase();
 
+  // Timeline/session signals
   if (/\b(last session|yesterday|prior session|previous session|last time we|handoff|what happened last|what did we do|cross.session|earlier today|what we discussed|when we last)\b/i.test(q)) {
     return "If searching memory for this: use session_log or memory_retrieve, NOT query.";
   }
 
+  // Causal signals
   if (/\b(why did|why was|why were|what caused|what led to|reason for|decided to|decision about|trade.?off|instead of|chose to)\b/i.test(q) || /^why\b/i.test(q)) {
     return "If searching memory for this: use intent_search or memory_retrieve, NOT query.";
   }
 
+  // Discovery signals
   if (/\b(similar to|related to|what else|what other|reminds? me of|like this)\b/i.test(q)) {
     return "If searching memory for this: use find_similar or memory_retrieve, NOT query.";
   }

package/src/hooks/curator-nudge.ts

@@ -9,6 +9,7 @@
 import { resolve as pathResolve } from "path";
 import { existsSync, readFileSync } from "fs";
 import type { Store } from "../store.ts";
+import { getIndexHealth } from "../store.ts";
 import type { HookInput, HookOutput } from "../hooks.ts";
 import {
   makeContextOutput,
@@ -64,11 +65,23 @@ export async function curatorNudge(
     return makeEmptyOutput("curator-nudge");
   }
 
+  // Override embedding backlog with live data (report value goes stale after embed timer runs)
+  let actions = [...report.actions];
+  try {
+    const health = getIndexHealth(store.db);
+    actions = actions.filter(a => !/documents? need embedding/i.test(a));
+    if (health.needsEmbedding > 0) {
+      actions.unshift(`${health.needsEmbedding} documents need embedding`);
+    }
+  } catch { /* fail-open: use report actions as-is */ }
+
+  if (actions.length === 0) return makeEmptyOutput("curator-nudge");
+
   // Build compact action summary within budget
   const lines = [`**Curator (${report.timestamp.slice(0, 10)}):**`];
   let tokens = estimateTokens(lines[0]!);
 
-  for (const action of report.actions) {
+  for (const action of actions) {
     const line = `- ${action}`;
     const lineTokens = estimateTokens(line);
     if (tokens + lineTokens > MAX_TOKEN_BUDGET && lines.length > 1) break;