clawmem 0.15.1 → 0.16.0

package/AGENTS.md

@@ -174,7 +174,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
 - **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed`.
 - **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs`. Don't run it after every reindex (A-MEM links per-doc automatically).
 - **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert scores, silently collapsing ranking to RRF.
-- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → almost always the context-surfacing hook's **cold-start** (fresh Bun process + opening a large `index.sqlite` + cold OS page cache), NOT inference. Recurs on memory-constrained hosts (e.g. WSL with a low memory cap) as the cache is evicted between turns. **Durable fix: give the host enough RAM to keep the index cached**; raising the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) is a secondary margin — avoid 15s+ as a standing default. Detail: [docs/troubleshooting.md](docs/troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
+- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → **fixed in v0.16.0** (upgrade). Root cause was not inference or host RAM alone: the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. v0.16.0 bounds both vector legs with real deadlines + timer cleanup, read-guards the init backfill, caps the init `busy_timeout`, and adds a watcher-side vector prewarm. A cold OS page cache still adds latency to the first post-boot call, so RAM headroom helps the margin — but on a large vault the scan cost, not RAM, was the dominant trigger. Still seeing it after upgrading? Raise the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) as a secondary margin. Detail: [docs/troubleshooting.md](docs/troubleshooting.md).
 - **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [docs/troubleshooting.md](docs/troubleshooting.md).
 
 ---

package/docs/troubleshooting.md

@@ -136,7 +136,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
 - If the error persists after v0.1.8: restart the watcher to clear accumulated state (`systemctl --user restart clawmem-watcher.service`). Check `systemctl --user status clawmem-watcher.service` for memory usage — healthy is under 100MB, bloated is 400MB+.
 - **v0.2.4 fix:** Hook's SQLite `busy_timeout` was 500ms — too tight. During A-MEM enrichment or heavy indexing, the watcher can hold write locks for 500ms+, causing the hook's DB open to fail with SQLITE_BUSY. Raised to 5000ms (matches MCP server). The hook's 8s outer timeout still leaves 3s for actual work after a 5s busy wait.
 - **v0.3.1 fix:** Shell `timeout` wrappers (e.g., `timeout 8 clawmem hook context-surfacing`) kill the process with exit 124 and no stderr — Claude Code reports "Failed with non-blocking status code: No stderr output". This affects all hook events (UserPromptSubmit, Stop, SessionStart, PreCompact), not just Stop hooks. Fix: Remove shell `timeout` from all hook commands and use Claude Code's native `timeout` property instead. Run `clawmem setup hooks` to reinstall with correct config (v0.3.1+), or manually update `~/.claude/settings.json` — see [setup-hooks](guides/setup-hooks.md).
-- **Large vault + memory-constrained host (intermittent cold-start, even with `llama-server`).** A correctly-configured `llama-server` (the ~200ms row under *Hooks slow or near timeout*) can still intermittently exceed the 8s budget on the **first/cold call of a session — or any later turn after the OS evicts the page cache**. The cost here is not inference and not the `node-llama-cpp` import; it is process + data cold-start: spawning a fresh Bun process, opening a large `index.sqlite` (this grows with vault size — a multi-hundred-MB to >1GB index is the trigger), and re-reading index pages dropped from the OS page cache. On a memory-constrained host this *recurs* — e.g. a WSL2 instance with a low `.wslconfig` `memory` cap (or any box under memory pressure) evicts the cached index + Bun modules between turns, so "cold start" is no longer once-per-session. Warm calls stay sub-second; only the cold ones blow the budget, which is why it presents as intermittent "timed out after 8s" on *certain* turns. **Durable fix: give the host enough RAM that the index + Bun modules stay cached** (e.g. raise the WSL2 `memory` cap) — this removes the recurrence at the source. A modest `timeout` bump (see the tradeoffs table under *Hooks slow or near timeout*) is a secondary margin for the genuine cold call, not a substitute for memory headroom. The `deep` profile additionally reranks (extra remote round-trips), widening the cold-call window; `balanced` (default) does not rerank.
+- **Large vault + intermittent hook timeout (`timed out after 8s`) — FIXED in v0.16.0.** Earlier this was diagnosed as pure cold-start (fresh Bun process, opening a large `index.sqlite`, re-reading evicted index pages) with "give the host more RAM" as the durable fix — but the dominant causes were two code-level defects: (1) the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and (2) every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. **v0.16.0 fixes both:** `searchVec` takes a real wall-clock deadline and self-aborts before the blocking scan; both vector legs race the embed against the remaining budget and clear their timers; the init backfill is read-guarded and the init `busy_timeout` is capped to the caller's value; and the watcher prewarms the sqlite-vec payload into the page cache on startup (embed-independent, watcher-only). A cold page cache still adds latency to the genuine first post-boot call, so host RAM headroom + the prewarm help the margin — but on a large vault the scan cost, not RAM, was the trigger. A modest `timeout` bump (see the tradeoffs table under *Hooks slow or near timeout*) remains a secondary margin. The `deep` profile additionally reranks (extra remote round-trips), widening the cold-call window; `balanced` (default) does not rerank.
 
 **Watcher memory bloat (400MB+)**
 - The watcher accumulates memory when processing high-frequency file change events. The most common trigger was Claude Code session transcript `.jsonl` files changing on every keystroke during active conversations. Each event opened the database briefly, and over hours of active use, memory grew to 400-800MB.
@@ -222,7 +222,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
 
   The timeout applies per invocation. A slow first prompt (cold start) doesn't mean subsequent prompts will be slow — Bun caches modules after the first load, and `node-llama-cpp` model files are cached on disk after the first download. Subsequent prompts in the same session are typically faster.
 
-  **Exception — memory-constrained hosts / very large vaults:** if the host is under memory pressure (e.g. a WSL2 instance with a low `.wslconfig` `memory` cap) or the vault's `index.sqlite` is large, the OS can evict the cached index + Bun modules *between* turns, so the cold-start cost recurs intermittently rather than once per session — the hook times out on *certain* turns, not just the first. There the durable fix is host memory headroom (so the working set stays cached), not a longer timeout. See *"UserPromptSubmit hook error" (intermittent)* above.
+  **Exception — large vaults (intermittent, pre-v0.16.0):** before v0.16.0 the hook could time out on *certain* turns (not just the first) because of an unbounded synchronous `sqlite-vec` scan plus an init-time write-lock wait — see *"UserPromptSubmit hook error" (intermittent)* above. **Upgrade to v0.16.0**, which bounds the scan and the init path and prewarms the cache. Host RAM headroom + the watcher prewarm still help the genuine cold-call margin, but they were not the root cause.
 
   **Stop hooks** (`decision-extractor`, `handoff-generator`, `feedback-loop`) default to 30s (v0.3.1+, was 10s prior) because they run LLM inference (observer model). These run at session end, so latency doesn't block the user.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clawmem",
-  "version": "0.15.1",
+  "version": "0.16.0",
   "description": "On-device memory layer for AI agents. Claude Code, OpenClaw, and Hermes. Hooks + MCP server + hybrid RAG search.",
   "type": "module",
   "bin": {

package/src/clawmem.ts

@@ -8,6 +8,7 @@ import { existsSync, mkdirSync, readFileSync } from "fs";
 import { resolve as pathResolve, basename } from "path";
 import {
   createStore,
+  prewarmVectors,
   enableProductionMode,
   getDefaultDbPath,
   canonicalDocId,
@@ -1886,6 +1887,20 @@ async function cmdWatch() {
     stopHeavyLane = startHeavyMaintenanceWorker(s, llm, cfg);
   }
 
+  // Prewarm the sqlite-vec chunks into OS page cache ONCE, in the single long-lived watcher
+  // process only (never in the per-session MCP processes — N concurrent cold scans would be an
+  // I/O storm). The context-surfacing UserPromptSubmit hook runs a SYNCHRONOUS sqlite-vec MATCH
+  // that cannot be time-bounded in-thread (bun:sqlite exposes no interrupt); a cold ~1.5 GB scan
+  // can blow the hook's 8-15s budget. A single warm scan here keeps the hook-path scan sub-second.
+  // Deferred + best-effort so it never delays watcher startup and never throws.
+  setTimeout(() => {
+    try {
+      // prewarmVectors is embed-independent and returns true ONLY when a scan actually ran,
+      // so we never log a false-positive "prewarmed" (embed down at boot / no vectors yet).
+      if (prewarmVectors(s.db)) console.log(`${c.dim}[watch] vector cache prewarmed${c.reset}`);
+    } catch { /* best-effort: unexpected SQL error */ }
+  }, 0);
+
   watcherHandle = startWatcher(dirs, {
     debounceMs: 2000,
     onChanged: async (fullPath, event) => {

package/src/hooks/context-surfacing.ts

@@ -183,14 +183,24 @@ export async function contextSurfacing(
   // When vector succeeds, also supplement with FTS for keyword-exact recall
   let results: SearchResult[] = [];
   if (profile.useVector) {
+    let vectorTimer: ReturnType<typeof setTimeout> | undefined;
     try {
-      const vectorPromise = store.searchVec(retrievalQuery, DEFAULT_EMBED_MODEL, maxResults);
-      const timeoutPromise = new Promise<SearchResult[]>((_, reject) =>
-        setTimeout(() => reject(new Error("vector timeout")), profile.vectorTimeout)
-      );
+      // Pass a wall-clock deadline into searchVec: the Promise.race below abandons the vector
+      // promise on timeout but cannot CANCEL it (and cannot interrupt its synchronous scan). The
+      // deadline makes searchVec self-abort before the blocking MATCH, so a slow embed cannot let
+      // an already-timed-out vector leg resume and re-block the hook after it fell back to FTS.
+      const vectorDeadline = Date.now() + profile.vectorTimeout;
+      const vectorPromise = store.searchVec(retrievalQuery, DEFAULT_EMBED_MODEL, maxResults, undefined, undefined, undefined, vectorDeadline);
+      const timeoutPromise = new Promise<SearchResult[]>((_, reject) => {
+        vectorTimer = setTimeout(() => reject(new Error("vector timeout")), profile.vectorTimeout);
+      });
       results = await Promise.race([vectorPromise, timeoutPromise]);
     } catch {
       // Vector search unavailable, timed out, or errored — fall back to BM25
+    } finally {
+      // Clear the timer when the vector promise won the race: a pending (ref'd) setTimeout keeps the
+      // Bun hook process alive for the full vectorTimeout after results are already in hand.
+      if (vectorTimer) clearTimeout(vectorTimer);
     }
   }
 
@@ -269,7 +279,21 @@ export async function contextSurfacing(
             if (eq.type === 'lex') {
               hits = store.searchFTS(eq.query, 5);
             } else if (profile.useVector) {
-              try { hits = await store.searchVec(eq.query, DEFAULT_EMBED_MODEL, 5); } catch { /* vector leg non-fatal */ }
+              // Bound BOTH the async embed wait (Promise.race on the remaining 6s budget) AND the
+              // late synchronous MATCH (the deadline arg makes the abandoned promise self-abort
+              // before the scan) — mirroring the balanced leg above. The loop guard only breaks
+              // BETWEEN iterations, so without the race a slow embed here can still blow the budget.
+              const remainingMs = startTime + 6000 - Date.now();
+              if (remainingMs <= 0) break;
+              let deepTimer: ReturnType<typeof setTimeout> | undefined;
+              try {
+                const deepVec = store.searchVec(eq.query, DEFAULT_EMBED_MODEL, 5, undefined, undefined, undefined, startTime + 6000);
+                const deepTimeout = new Promise<SearchResult[]>((_, reject) => {
+                  deepTimer = setTimeout(() => reject(new Error("vector timeout")), remainingMs);
+                });
+                hits = await Promise.race([deepVec, deepTimeout]);
+              } catch { /* vector leg non-fatal (timed out or errored) */ }
+              finally { if (deepTimer) clearTimeout(deepTimer); }  // don't let a pending timer keep the hook process alive
             }
             for (const r of hits) {
               if (!seen.has(r.filepath)) {

package/src/store.ts

@@ -365,7 +365,7 @@ function loadVecExtension(db: Database): void {
   }
 }
 
-function initializeDatabase(db: Database): void {
+function initializeDatabase(db: Database, busyTimeoutMs: number = 15000): void {
   // Set busy_timeout FIRST so subsequent PRAGMAs (journal_mode in particular,
   // which acquires a write lock when switching or initializing WAL state) wait
   // instead of returning SQLITE_BUSY when concurrent Stop hooks
@@ -373,10 +373,12 @@ function initializeDatabase(db: Database): void {
   // before_reset hook fan-out in src/openclaw/engine.ts — open the DB
   // simultaneously. busy_timeout is a connection-level setting that only
   // governs *subsequent* statements (default busy handler is NULL → SQLITE_BUSY
-  // returns immediately), so it must precede the contending PRAGMAs. 15s is
-  // well within the 30s Stop hook timeout. createStore() resets to operational
+  // returns immediately), so it must precede the contending PRAGMAs. The init
+  // busy_timeout defaults to 15s (well within the 30s Stop hook timeout) but is
+  // capped to the caller's opts.busyTimeout — hook opens pass 5000 so init cannot
+  // wait out the 8-15s UserPromptSubmit budget. createStore() resets to operational
   // value (5000ms or opts.busyTimeout) after DDL completes. Issue #13.
-  db.exec("PRAGMA busy_timeout = 15000");
+  db.exec(`PRAGMA busy_timeout = ${busyTimeoutMs}`);
   loadVecExtension(db);
   db.exec("PRAGMA journal_mode = WAL");
   db.exec("PRAGMA foreign_keys = ON");
@@ -450,9 +452,17 @@ function initializeDatabase(db: Database): void {
     }
   }
 
-  // Backfill last_accessed_at from modified_at for existing docs
+  // Backfill last_accessed_at from modified_at for existing docs.
+  // Guarded by a read first: on an already-backfilled DB the UPDATE is skipped, so a writable
+  // open takes NO write lock here and cannot wait on busy_timeout under concurrent writers.
+  // (The unconditional UPDATE previously ran on EVERY writable open — including the
+  // context-surfacing UserPromptSubmit hook — and could block up to busy_timeout when another
+  // process held the write lock, pushing the hook past its 8-15s deadline.)
   try {
-    db.exec(`UPDATE documents SET last_accessed_at = modified_at WHERE last_accessed_at IS NULL`);
+    const needsBackfill = db.prepare(`SELECT 1 FROM documents WHERE last_accessed_at IS NULL LIMIT 1`).get();
+    if (needsBackfill) {
+      db.exec(`UPDATE documents SET last_accessed_at = modified_at WHERE last_accessed_at IS NULL`);
+    }
   } catch { /* ignore if already backfilled */ }
 
   db.exec(`CREATE INDEX IF NOT EXISTS idx_documents_collection ON documents(collection, active)`);
@@ -1136,6 +1146,22 @@ export function getVecTableDim(db: Database): number | null {
   return readVecTableDim(db);
 }
 
+/**
+ * Prewarm the sqlite-vec payload into OS page cache with a single brute-force MATCH using a ZERO
+ * query vector. Decoupled from the embedding server on purpose — it works when the embed server is
+ * down at boot or CLAWMEM_NO_LOCAL_MODELS=true, unlike a searchVec()-based prewarm (which embeds
+ * first and would silently no-op). Returns true ONLY if a scan actually ran (a vector table with a
+ * known dimension exists), so callers never log a false-positive "warmed". The k-NN MATCH is
+ * brute-force, so it touches every vector chunk — exactly the payload we want cache-resident.
+ */
+export function prewarmVectors(db: Database): boolean {
+  const dim = getVecTableDim(db);
+  if (!dim || dim <= 0) return false;
+  const zero = new Float32Array(dim);
+  db.prepare(`SELECT hash_seq FROM vectors_vec WHERE embedding MATCH ? AND k = ?`).all(zero, 1);
+  return true;
+}
+
 /**
  * The DISTINCT non-empty embedding models stored in the vault (empty array if no
  * embeddings exist). Used to detect model drift BETWEEN runs — a different model at
@@ -1204,7 +1230,7 @@ export type Store = {
 
   // Search
   searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => SearchResult[];
-  searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => Promise<SearchResult[]>;
+  searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => Promise<SearchResult[]>;
 
   // Query expansion & reranking
   expandQuery: (query: string, model?: string, intent?: string) => Promise<ExpandedQuery[]>;
@@ -1339,7 +1365,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
     ? new Database(resolvedPath, { readonly: true })
     : new Database(resolvedPath);
   if (!opts?.readonly) {
-    initializeDatabase(db);
+    initializeDatabase(db, opts?.busyTimeout ?? 15000);
   } else {
     // Readonly: set busy_timeout FIRST so the journal_mode PRAGMA below
     // doesn't race when concurrent processes open the DB. PRAGMA
@@ -1352,7 +1378,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
     db.exec("PRAGMA journal_mode = WAL");
     db.exec("PRAGMA query_only = ON");
   }
-  // For the writable branch: initializeDatabase() set 15000 during DDL —
+  // For the writable branch: initializeDatabase() set opts.busyTimeout (default 15000) during DDL —
   // reset to operational value here. For readonly: already set inside the
   // branch above; this assignment is a no-op rewrite to the same value.
   db.exec(`PRAGMA busy_timeout = ${opts?.busyTimeout ?? 5000}`);
@@ -1398,7 +1424,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
 
     // Search
     searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchFTS(db, query, limit, collectionId, collections, dateRange),
-    searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchVec(db, query, model, limit, collectionId, collections, dateRange),
+    searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => searchVec(db, query, model, limit, collectionId, collections, dateRange, deadlineMs),
 
     // Query expansion & reranking
     expandQuery: (query: string, model?: string, intent?: string) => expandQuery(query, model, db, intent),
@@ -3361,13 +3387,19 @@ export function searchFTS(db: Database, query: string, limit: number = 20, colle
 // Vector Search
 // =============================================================================
 
-export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }): Promise<SearchResult[]> {
+export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number): Promise<SearchResult[]> {
   const tableExists = db.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name='vectors_vec'`).get();
   if (!tableExists) return [];
 
   const embedding = await getEmbedding(query, model, true);
   if (!embedding) return [];
 
+  // Guard-defect fix: the caller's Promise.race(vectorTimeout) cannot interrupt the SYNCHRONOUS
+  // sqlite-vec MATCH below (bun:sqlite blocks the event loop) and does NOT cancel this promise.
+  // If the wall-clock budget already elapsed during the async embed above, bail here so a
+  // timed-out vector leg cannot resume and re-block the hook after it fell back to FTS.
+  if (deadlineMs !== undefined && Date.now() >= deadlineMs) return [];
+
   // IMPORTANT: We use a two-step query approach here because sqlite-vec virtual tables
   // hang indefinitely when combined with JOINs in the same query. Do NOT try to
   // "optimize" this by combining into a single query with JOINs - it will break.