clawmem 0.15.0 → 0.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -174,7 +174,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
174
174
  - **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed`.
175
175
  - **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs`. Don't run it after every reindex (A-MEM links per-doc automatically).
176
176
  - **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert scores, silently collapsing ranking to RRF.
177
- - **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → almost always the context-surfacing hook's **cold-start** (fresh Bun process + opening a large `index.sqlite` + cold OS page cache), NOT inference. Recurs on memory-constrained hosts (e.g. WSL with a low memory cap) as the cache is evicted between turns. **Durable fix: give the host enough RAM to keep the index cached**; raising the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) is a secondary margin — avoid 15s+ as a standing default. Detail: [docs/troubleshooting.md](docs/troubleshooting.md) → *Tuning the context-surfacing hook timeout*.
177
+ - **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → **fixed in v0.16.0** (upgrade). Root cause was not inference or host RAM alone: the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. v0.16.0 bounds both vector legs with real deadlines + timer cleanup, read-guards the init backfill, caps the init `busy_timeout`, and adds a watcher-side vector prewarm. A cold OS page cache still adds latency to the first post-boot call, so RAM headroom helps the margin but on a large vault the scan cost, not RAM, was the dominant trigger. Still seeing it after upgrading? Raise the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) as a secondary margin. Detail: [docs/troubleshooting.md](docs/troubleshooting.md).
178
178
  - **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [docs/troubleshooting.md](docs/troubleshooting.md).
179
179
 
180
180
  ---
@@ -12,6 +12,15 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
12
12
  - If both snap bun (`/snap/bin/bun`) and native bun (`~/.bun/bin/bun`) are installed, `which bun` may return the snap version. Direct `bun -e` or `bun run` commands will use the wrong binary.
13
13
  - Fix: The `bin/clawmem` wrapper handles this automatically. For manual commands, use `~/.bun/bin/bun` explicitly or add `~/.bun/bin` to PATH before `/snap/bin`.
14
14
 
15
+ **macOS: `bootstrap` / `doctor` fails with "does not support dynamic extension loading" (sqlite-vec on macOS)**
16
+ - `clawmem bootstrap` (or `clawmem doctor`) fails at the database step because macOS's built-in SQLite — which Bun uses by default — is compiled without extension-loading support, so the `sqlite-vec` vector extension cannot load. Symptom: `✗ Database: ... This build of sqlite3 does not support dynamic extension loading`. Yoloshii/ClawMem#20.
17
+ - Fix: install an extension-capable SQLite via Homebrew, then re-run bootstrap:
18
+ ```bash
19
+ brew install sqlite
20
+ clawmem bootstrap ~/notes --name notes
21
+ ```
22
+ - ClawMem auto-detects Homebrew's SQLite at the standard prefixes (`/opt/homebrew` on Apple Silicon, `/usr/local` on Intel) and at `brew --prefix sqlite` for non-standard prefixes — installing it is all that's required, no env var or config. If it still fails after `brew install sqlite`, run `brew reinstall sqlite` and confirm `ls $(brew --prefix sqlite)/lib/libsqlite3.dylib` resolves.
23
+
15
24
  ## Embedding & GPU
16
25
 
17
26
  **"Local model download blocked" error**
@@ -127,7 +136,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
127
136
  - If the error persists after v0.1.8: restart the watcher to clear accumulated state (`systemctl --user restart clawmem-watcher.service`). Check `systemctl --user status clawmem-watcher.service` for memory usage — healthy is under 100MB, bloated is 400MB+.
128
137
  - **v0.2.4 fix:** Hook's SQLite `busy_timeout` was 500ms — too tight. During A-MEM enrichment or heavy indexing, the watcher can hold write locks for 500ms+, causing the hook's DB open to fail with SQLITE_BUSY. Raised to 5000ms (matches MCP server). The hook's 8s outer timeout still leaves 3s for actual work after a 5s busy wait.
129
138
  - **v0.3.1 fix:** Shell `timeout` wrappers (e.g., `timeout 8 clawmem hook context-surfacing`) kill the process with exit 124 and no stderr — Claude Code reports "Failed with non-blocking status code: No stderr output". This affects all hook events (UserPromptSubmit, Stop, SessionStart, PreCompact), not just Stop hooks. Fix: Remove shell `timeout` from all hook commands and use Claude Code's native `timeout` property instead. Run `clawmem setup hooks` to reinstall with correct config (v0.3.1+), or manually update `~/.claude/settings.json` — see [setup-hooks](guides/setup-hooks.md).
130
- - **Large vault + memory-constrained host (intermittent cold-start, even with `llama-server`).** A correctly-configured `llama-server` (the ~200ms row under *Hooks slow or near timeout*) can still intermittently exceed the 8s budget on the **first/cold call of a session or any later turn after the OS evicts the page cache**. The cost here is not inference and not the `node-llama-cpp` import; it is process + data cold-start: spawning a fresh Bun process, opening a large `index.sqlite` (this grows with vault size a multi-hundred-MB to >1GB index is the trigger), and re-reading index pages dropped from the OS page cache. On a memory-constrained host this *recurs* e.g. a WSL2 instance with a low `.wslconfig` `memory` cap (or any box under memory pressure) evicts the cached index + Bun modules between turns, so "cold start" is no longer once-per-session. Warm calls stay sub-second; only the cold ones blow the budget, which is why it presents as intermittent "timed out after 8s" on *certain* turns. **Durable fix: give the host enough RAM that the index + Bun modules stay cached** (e.g. raise the WSL2 `memory` cap)this removes the recurrence at the source. A modest `timeout` bump (see the tradeoffs table under *Hooks slow or near timeout*) is a secondary margin for the genuine cold call, not a substitute for memory headroom. The `deep` profile additionally reranks (extra remote round-trips), widening the cold-call window; `balanced` (default) does not rerank.
139
+ - **Large vault + intermittent hook timeout (`timed out after 8s`) FIXED in v0.16.0.** Earlier this was diagnosed as pure cold-start (fresh Bun process, opening a large `index.sqlite`, re-reading evicted index pages) with "give the host more RAM" as the durable fix but the dominant causes were two code-level defects: (1) the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and (2) every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. **v0.16.0 fixes both:** `searchVec` takes a real wall-clock deadline and self-aborts before the blocking scan; both vector legs race the embed against the remaining budget and clear their timers; the init backfill is read-guarded and the init `busy_timeout` is capped to the caller's value; and the watcher prewarms the sqlite-vec payload into the page cache on startup (embed-independent, watcher-only). A cold page cache still adds latency to the genuine first post-boot call, so host RAM headroom + the prewarm help the margin but on a large vault the scan cost, not RAM, was the trigger. A modest `timeout` bump (see the tradeoffs table under *Hooks slow or near timeout*) remains a secondary margin. The `deep` profile additionally reranks (extra remote round-trips), widening the cold-call window; `balanced` (default) does not rerank.
131
140
 
132
141
  **Watcher memory bloat (400MB+)**
133
142
  - The watcher accumulates memory when processing high-frequency file change events. The most common trigger was Claude Code session transcript `.jsonl` files changing on every keystroke during active conversations. Each event opened the database briefly, and over hours of active use, memory grew to 400-800MB.
@@ -213,7 +222,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
213
222
 
214
223
  The timeout applies per invocation. A slow first prompt (cold start) doesn't mean subsequent prompts will be slow — Bun caches modules after the first load, and `node-llama-cpp` model files are cached on disk after the first download. Subsequent prompts in the same session are typically faster.
215
224
 
216
- **Exception — memory-constrained hosts / very large vaults:** if the host is under memory pressure (e.g. a WSL2 instance with a low `.wslconfig` `memory` cap) or the vault's `index.sqlite` is large, the OS can evict the cached index + Bun modules *between* turns, so the cold-start cost recurs intermittently rather than once per session — the hook times out on *certain* turns, not just the first. There the durable fix is host memory headroom (so the working set stays cached), not a longer timeout. See *"UserPromptSubmit hook error" (intermittent)* above.
225
+ **Exception — large vaults (intermittent, pre-v0.16.0):** before v0.16.0 the hook could time out on *certain* turns (not just the first) because of an unbounded synchronous `sqlite-vec` scan plus an init-time write-lock wait see *"UserPromptSubmit hook error" (intermittent)* above. **Upgrade to v0.16.0**, which bounds the scan and the init path and prewarms the cache. Host RAM headroom + the watcher prewarm still help the genuine cold-call margin, but they were not the root cause.
217
226
 
218
227
  **Stop hooks** (`decision-extractor`, `handoff-generator`, `feedback-loop`) default to 30s (v0.3.1+, was 10s prior) because they run LLM inference (observer model). These run at session end, so latency doesn't block the user.
219
228
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "clawmem",
3
- "version": "0.15.0",
3
+ "version": "0.16.0",
4
4
  "description": "On-device memory layer for AI agents. Claude Code, OpenClaw, and Hermes. Hooks + MCP server + hybrid RAG search.",
5
5
  "type": "module",
6
6
  "bin": {
package/src/clawmem.ts CHANGED
@@ -8,6 +8,7 @@ import { existsSync, mkdirSync, readFileSync } from "fs";
8
8
  import { resolve as pathResolve, basename } from "path";
9
9
  import {
10
10
  createStore,
11
+ prewarmVectors,
11
12
  enableProductionMode,
12
13
  getDefaultDbPath,
13
14
  canonicalDocId,
@@ -1886,6 +1887,20 @@ async function cmdWatch() {
1886
1887
  stopHeavyLane = startHeavyMaintenanceWorker(s, llm, cfg);
1887
1888
  }
1888
1889
 
1890
+ // Prewarm the sqlite-vec chunks into OS page cache ONCE, in the single long-lived watcher
1891
+ // process only (never in the per-session MCP processes — N concurrent cold scans would be an
1892
+ // I/O storm). The context-surfacing UserPromptSubmit hook runs a SYNCHRONOUS sqlite-vec MATCH
1893
+ // that cannot be time-bounded in-thread (bun:sqlite exposes no interrupt); a cold ~1.5 GB scan
1894
+ // can blow the hook's 8-15s budget. A single warm scan here keeps the hook-path scan sub-second.
1895
+ // Deferred + best-effort so it never delays watcher startup and never throws.
1896
+ setTimeout(() => {
1897
+ try {
1898
+ // prewarmVectors is embed-independent and returns true ONLY when a scan actually ran,
1899
+ // so we never log a false-positive "prewarmed" (embed down at boot / no vectors yet).
1900
+ if (prewarmVectors(s.db)) console.log(`${c.dim}[watch] vector cache prewarmed${c.reset}`);
1901
+ } catch { /* best-effort: unexpected SQL error */ }
1902
+ }, 0);
1903
+
1889
1904
  watcherHandle = startWatcher(dirs, {
1890
1905
  debounceMs: 2000,
1891
1906
  onChanged: async (fullPath, event) => {
@@ -183,14 +183,24 @@ export async function contextSurfacing(
183
183
  // When vector succeeds, also supplement with FTS for keyword-exact recall
184
184
  let results: SearchResult[] = [];
185
185
  if (profile.useVector) {
186
+ let vectorTimer: ReturnType<typeof setTimeout> | undefined;
186
187
  try {
187
- const vectorPromise = store.searchVec(retrievalQuery, DEFAULT_EMBED_MODEL, maxResults);
188
- const timeoutPromise = new Promise<SearchResult[]>((_, reject) =>
189
- setTimeout(() => reject(new Error("vector timeout")), profile.vectorTimeout)
190
- );
188
+ // Pass a wall-clock deadline into searchVec: the Promise.race below abandons the vector
189
+ // promise on timeout but cannot CANCEL it (and cannot interrupt its synchronous scan). The
190
+ // deadline makes searchVec self-abort before the blocking MATCH, so a slow embed cannot let
191
+ // an already-timed-out vector leg resume and re-block the hook after it fell back to FTS.
192
+ const vectorDeadline = Date.now() + profile.vectorTimeout;
193
+ const vectorPromise = store.searchVec(retrievalQuery, DEFAULT_EMBED_MODEL, maxResults, undefined, undefined, undefined, vectorDeadline);
194
+ const timeoutPromise = new Promise<SearchResult[]>((_, reject) => {
195
+ vectorTimer = setTimeout(() => reject(new Error("vector timeout")), profile.vectorTimeout);
196
+ });
191
197
  results = await Promise.race([vectorPromise, timeoutPromise]);
192
198
  } catch {
193
199
  // Vector search unavailable, timed out, or errored — fall back to BM25
200
+ } finally {
201
+ // Clear the timer when the vector promise won the race: a pending (ref'd) setTimeout keeps the
202
+ // Bun hook process alive for the full vectorTimeout after results are already in hand.
203
+ if (vectorTimer) clearTimeout(vectorTimer);
194
204
  }
195
205
  }
196
206
 
@@ -269,7 +279,21 @@ export async function contextSurfacing(
269
279
  if (eq.type === 'lex') {
270
280
  hits = store.searchFTS(eq.query, 5);
271
281
  } else if (profile.useVector) {
272
- try { hits = await store.searchVec(eq.query, DEFAULT_EMBED_MODEL, 5); } catch { /* vector leg non-fatal */ }
282
+ // Bound BOTH the async embed wait (Promise.race on the remaining 6s budget) AND the
283
+ // late synchronous MATCH (the deadline arg makes the abandoned promise self-abort
284
+ // before the scan) — mirroring the balanced leg above. The loop guard only breaks
285
+ // BETWEEN iterations, so without the race a slow embed here can still blow the budget.
286
+ const remainingMs = startTime + 6000 - Date.now();
287
+ if (remainingMs <= 0) break;
288
+ let deepTimer: ReturnType<typeof setTimeout> | undefined;
289
+ try {
290
+ const deepVec = store.searchVec(eq.query, DEFAULT_EMBED_MODEL, 5, undefined, undefined, undefined, startTime + 6000);
291
+ const deepTimeout = new Promise<SearchResult[]>((_, reject) => {
292
+ deepTimer = setTimeout(() => reject(new Error("vector timeout")), remainingMs);
293
+ });
294
+ hits = await Promise.race([deepVec, deepTimeout]);
295
+ } catch { /* vector leg non-fatal (timed out or errored) */ }
296
+ finally { if (deepTimer) clearTimeout(deepTimer); } // don't let a pending timer keep the hook process alive
273
297
  }
274
298
  for (const r of hits) {
275
299
  if (!seen.has(r.filepath)) {
package/src/store.ts CHANGED
@@ -14,7 +14,7 @@
14
14
 
15
15
  import { Database } from "bun:sqlite";
16
16
  import { Glob } from "bun";
17
- import { realpathSync } from "node:fs";
17
+ import { realpathSync, existsSync } from "node:fs";
18
18
  import * as sqliteVec from "sqlite-vec";
19
19
  import {
20
20
  LlamaCpp,
@@ -290,17 +290,82 @@ export function toVirtualPath(db: Database, absolutePath: string): string | null
290
290
  // Database initialization
291
291
  // =============================================================================
292
292
 
293
- // On macOS, use Homebrew's SQLite which supports extensions
293
+ // On macOS, Apple's built-in libsqlite3 — which Bun uses by default — is
294
+ // compiled WITHOUT extension-loading support, so sqliteVec.load() fails with
295
+ // "This build of sqlite3 does not support dynamic extension loading" (Issue #20).
296
+ // Point Bun at an extension-capable SQLite (Homebrew's) via setCustomSQLite()
297
+ // BEFORE the first Database is opened. setCustomSQLite() must receive a path
298
+ // that EXISTS — an invalid path hard-crashes Bun (oven-sh/bun#18811) — so every
299
+ // candidate is existence-checked first. The resolved path (or null) is recorded
300
+ // so loadVecExtension() can emit an actionable error when no extension-capable
301
+ // SQLite is installed, instead of the cryptic extension-loading failure.
302
+
303
+ /** macOS only: the extension-capable SQLite activated via setCustomSQLite, or null if none was found. */
304
+ let macosCustomSqlitePath: string | null = null;
305
+
294
306
  if (process.platform === "darwin") {
295
- const homebrewSqlitePath = "/opt/homebrew/opt/sqlite/lib/libsqlite3.dylib";
307
+ const candidates = [
308
+ "/opt/homebrew/opt/sqlite/lib/libsqlite3.dylib", // Homebrew (Apple Silicon)
309
+ "/usr/local/opt/sqlite/lib/libsqlite3.dylib", // Homebrew (Intel)
310
+ ];
311
+ // For a non-standard Homebrew prefix, ask brew directly — but only when the
312
+ // standard paths are absent, so the common case pays no subprocess cost.
313
+ if (!candidates.some(p => existsSync(p))) {
314
+ try {
315
+ const brew = Bun.spawnSync(["brew", "--prefix", "sqlite"], { stdout: "pipe", stderr: "ignore" });
316
+ const prefix = brew.success ? brew.stdout.toString().trim() : "";
317
+ if (prefix) candidates.push(`${prefix}/lib/libsqlite3.dylib`);
318
+ } catch { /* brew not installed — nothing more to probe */ }
319
+ }
320
+ for (const candidate of candidates) {
321
+ if (!existsSync(candidate)) continue;
322
+ try {
323
+ Database.setCustomSQLite(candidate);
324
+ macosCustomSqlitePath = candidate;
325
+ break;
326
+ } catch { /* not usable — try the next candidate */ }
327
+ }
328
+ }
329
+
330
+ /**
331
+ * Translate a sqlite-vec load failure into an actionable error on macOS, where
332
+ * the default system SQLite cannot load extensions (Issue #20). Pure + exported
333
+ * for tests: pass `platform` / `foundPath` explicitly to exercise either branch.
334
+ * Non-macOS, or any unrelated error, is returned unchanged.
335
+ */
336
+ export function explainVecLoadError(
337
+ err: unknown,
338
+ platform: string = process.platform,
339
+ foundPath: string | null = macosCustomSqlitePath,
340
+ ): Error {
341
+ const original = err instanceof Error ? err : new Error(String(err));
342
+ const isExtensionError = /does not support dynamic extension loading/i.test(original.message);
343
+ if (platform !== "darwin" || !isExtensionError) return original;
344
+
345
+ const detail = foundPath === null
346
+ ? "No Homebrew SQLite was found at the standard locations (/opt/homebrew or /usr/local)."
347
+ : `A custom SQLite was set from ${foundPath}, but it still cannot load extensions — try 'brew reinstall sqlite'.`;
348
+ return new Error(
349
+ "ClawMem could not load the sqlite-vec extension: macOS's built-in SQLite is " +
350
+ "compiled without extension support.\n" +
351
+ "Fix: install an extension-capable SQLite with Homebrew, then re-run:\n" +
352
+ " brew install sqlite\n" +
353
+ `${detail}\n` +
354
+ "More detail: docs/troubleshooting.md (\"Bun runtime\" -> sqlite-vec on macOS), Yoloshii/ClawMem#20.\n" +
355
+ `Original error: ${original.message}`,
356
+ );
357
+ }
358
+
359
+ /** Load the sqlite-vec extension, surfacing an actionable error on macOS (Issue #20). */
360
+ function loadVecExtension(db: Database): void {
296
361
  try {
297
- if (Bun.file(homebrewSqlitePath).size > 0) {
298
- Database.setCustomSQLite(homebrewSqlitePath);
299
- }
300
- } catch { }
362
+ sqliteVec.load(db);
363
+ } catch (err) {
364
+ throw explainVecLoadError(err);
365
+ }
301
366
  }
302
367
 
303
- function initializeDatabase(db: Database): void {
368
+ function initializeDatabase(db: Database, busyTimeoutMs: number = 15000): void {
304
369
  // Set busy_timeout FIRST so subsequent PRAGMAs (journal_mode in particular,
305
370
  // which acquires a write lock when switching or initializing WAL state) wait
306
371
  // instead of returning SQLITE_BUSY when concurrent Stop hooks
@@ -308,11 +373,13 @@ function initializeDatabase(db: Database): void {
308
373
  // before_reset hook fan-out in src/openclaw/engine.ts — open the DB
309
374
  // simultaneously. busy_timeout is a connection-level setting that only
310
375
  // governs *subsequent* statements (default busy handler is NULL → SQLITE_BUSY
311
- // returns immediately), so it must precede the contending PRAGMAs. 15s is
312
- // well within the 30s Stop hook timeout. createStore() resets to operational
376
+ // returns immediately), so it must precede the contending PRAGMAs. The init
377
+ // busy_timeout defaults to 15s (well within the 30s Stop hook timeout) but is
378
+ // capped to the caller's opts.busyTimeout — hook opens pass 5000 so init cannot
379
+ // wait out the 8-15s UserPromptSubmit budget. createStore() resets to operational
313
380
  // value (5000ms or opts.busyTimeout) after DDL completes. Issue #13.
314
- db.exec("PRAGMA busy_timeout = 15000");
315
- sqliteVec.load(db);
381
+ db.exec(`PRAGMA busy_timeout = ${busyTimeoutMs}`);
382
+ loadVecExtension(db);
316
383
  db.exec("PRAGMA journal_mode = WAL");
317
384
  db.exec("PRAGMA foreign_keys = ON");
318
385
 
@@ -385,9 +452,17 @@ function initializeDatabase(db: Database): void {
385
452
  }
386
453
  }
387
454
 
388
- // Backfill last_accessed_at from modified_at for existing docs
455
+ // Backfill last_accessed_at from modified_at for existing docs.
456
+ // Guarded by a read first: on an already-backfilled DB the UPDATE is skipped, so a writable
457
+ // open takes NO write lock here and cannot wait on busy_timeout under concurrent writers.
458
+ // (The unconditional UPDATE previously ran on EVERY writable open — including the
459
+ // context-surfacing UserPromptSubmit hook — and could block up to busy_timeout when another
460
+ // process held the write lock, pushing the hook past its 8-15s deadline.)
389
461
  try {
390
- db.exec(`UPDATE documents SET last_accessed_at = modified_at WHERE last_accessed_at IS NULL`);
462
+ const needsBackfill = db.prepare(`SELECT 1 FROM documents WHERE last_accessed_at IS NULL LIMIT 1`).get();
463
+ if (needsBackfill) {
464
+ db.exec(`UPDATE documents SET last_accessed_at = modified_at WHERE last_accessed_at IS NULL`);
465
+ }
391
466
  } catch { /* ignore if already backfilled */ }
392
467
 
393
468
  db.exec(`CREATE INDEX IF NOT EXISTS idx_documents_collection ON documents(collection, active)`);
@@ -1071,6 +1146,22 @@ export function getVecTableDim(db: Database): number | null {
1071
1146
  return readVecTableDim(db);
1072
1147
  }
1073
1148
 
1149
+ /**
1150
+ * Prewarm the sqlite-vec payload into OS page cache with a single brute-force MATCH using a ZERO
1151
+ * query vector. Decoupled from the embedding server on purpose — it works when the embed server is
1152
+ * down at boot or CLAWMEM_NO_LOCAL_MODELS=true, unlike a searchVec()-based prewarm (which embeds
1153
+ * first and would silently no-op). Returns true ONLY if a scan actually ran (a vector table with a
1154
+ * known dimension exists), so callers never log a false-positive "warmed". The k-NN MATCH is
1155
+ * brute-force, so it touches every vector chunk — exactly the payload we want cache-resident.
1156
+ */
1157
+ export function prewarmVectors(db: Database): boolean {
1158
+ const dim = getVecTableDim(db);
1159
+ if (!dim || dim <= 0) return false;
1160
+ const zero = new Float32Array(dim);
1161
+ db.prepare(`SELECT hash_seq FROM vectors_vec WHERE embedding MATCH ? AND k = ?`).all(zero, 1);
1162
+ return true;
1163
+ }
1164
+
1074
1165
  /**
1075
1166
  * The DISTINCT non-empty embedding models stored in the vault (empty array if no
1076
1167
  * embeddings exist). Used to detect model drift BETWEEN runs — a different model at
@@ -1139,7 +1230,7 @@ export type Store = {
1139
1230
 
1140
1231
  // Search
1141
1232
  searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => SearchResult[];
1142
- searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => Promise<SearchResult[]>;
1233
+ searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => Promise<SearchResult[]>;
1143
1234
 
1144
1235
  // Query expansion & reranking
1145
1236
  expandQuery: (query: string, model?: string, intent?: string) => Promise<ExpandedQuery[]>;
@@ -1274,7 +1365,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
1274
1365
  ? new Database(resolvedPath, { readonly: true })
1275
1366
  : new Database(resolvedPath);
1276
1367
  if (!opts?.readonly) {
1277
- initializeDatabase(db);
1368
+ initializeDatabase(db, opts?.busyTimeout ?? 15000);
1278
1369
  } else {
1279
1370
  // Readonly: set busy_timeout FIRST so the journal_mode PRAGMA below
1280
1371
  // doesn't race when concurrent processes open the DB. PRAGMA
@@ -1283,11 +1374,11 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
1283
1374
  // production caller in this repo currently passes readonly:true,
1284
1375
  // but the ordering invariant should hold regardless. Issue #13.
1285
1376
  db.exec(`PRAGMA busy_timeout = ${opts?.busyTimeout ?? 5000}`);
1286
- sqliteVec.load(db);
1377
+ loadVecExtension(db);
1287
1378
  db.exec("PRAGMA journal_mode = WAL");
1288
1379
  db.exec("PRAGMA query_only = ON");
1289
1380
  }
1290
- // For the writable branch: initializeDatabase() set 15000 during DDL —
1381
+ // For the writable branch: initializeDatabase() set opts.busyTimeout (default 15000) during DDL —
1291
1382
  // reset to operational value here. For readonly: already set inside the
1292
1383
  // branch above; this assignment is a no-op rewrite to the same value.
1293
1384
  db.exec(`PRAGMA busy_timeout = ${opts?.busyTimeout ?? 5000}`);
@@ -1333,7 +1424,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
1333
1424
 
1334
1425
  // Search
1335
1426
  searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchFTS(db, query, limit, collectionId, collections, dateRange),
1336
- searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchVec(db, query, model, limit, collectionId, collections, dateRange),
1427
+ searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => searchVec(db, query, model, limit, collectionId, collections, dateRange, deadlineMs),
1337
1428
 
1338
1429
  // Query expansion & reranking
1339
1430
  expandQuery: (query: string, model?: string, intent?: string) => expandQuery(query, model, db, intent),
@@ -3296,13 +3387,19 @@ export function searchFTS(db: Database, query: string, limit: number = 20, colle
3296
3387
  // Vector Search
3297
3388
  // =============================================================================
3298
3389
 
3299
- export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }): Promise<SearchResult[]> {
3390
+ export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number): Promise<SearchResult[]> {
3300
3391
  const tableExists = db.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name='vectors_vec'`).get();
3301
3392
  if (!tableExists) return [];
3302
3393
 
3303
3394
  const embedding = await getEmbedding(query, model, true);
3304
3395
  if (!embedding) return [];
3305
3396
 
3397
+ // Guard-defect fix: the caller's Promise.race(vectorTimeout) cannot interrupt the SYNCHRONOUS
3398
+ // sqlite-vec MATCH below (bun:sqlite blocks the event loop) and does NOT cancel this promise.
3399
+ // If the wall-clock budget already elapsed during the async embed above, bail here so a
3400
+ // timed-out vector leg cannot resume and re-block the hook after it fell back to FTS.
3401
+ if (deadlineMs !== undefined && Date.now() >= deadlineMs) return [];
3402
+
3306
3403
  // IMPORTANT: We use a two-step query approach here because sqlite-vec virtual tables
3307
3404
  // hang indefinitely when combined with JOINs in the same query. Do NOT try to
3308
3405
  // "optimize" this by combining into a single query with JOINs - it will break.