clawmem 0.15.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +1 -1
- package/docs/troubleshooting.md +11 -2
- package/package.json +1 -1
- package/src/clawmem.ts +15 -0
- package/src/hooks/context-surfacing.ts +29 -5
- package/src/store.ts +117 -20
package/AGENTS.md
CHANGED
|
@@ -174,7 +174,7 @@ compositeScore = (0.50·searchScore + 0.25·recencyScore + 0.25·confidenceScore
|
|
|
174
174
|
- **Vector search empty but BM25 works** → missing embeddings (the watcher indexes but does NOT embed). Run `clawmem embed`.
|
|
175
175
|
- **`intent_search` weak for WHY/ENTITY** → sparse graph. Run `build_graphs`. Don't run it after every reindex (A-MEM links per-doc automatically).
|
|
176
176
|
- **Rankings look RRF-flat / reranker suspect** → `clawmem rerank-health`. A mis-served reranker (e.g. a GGUF that drops the score head) returns HTTP 200 but inert scores, silently collapsing ranking to RRF.
|
|
177
|
-
- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** →
|
|
177
|
+
- **Intermittent `UserPromptSubmit hook timed out after 8s — output discarded`** → **fixed in v0.16.0** (upgrade). Root cause was not inference or host RAM alone: the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. v0.16.0 bounds both vector legs with real deadlines + timer cleanup, read-guards the init backfill, caps the init `busy_timeout`, and adds a watcher-side vector prewarm. A cold OS page cache still adds latency to the first post-boot call, so RAM headroom helps the margin — but on a large vault the scan cost, not RAM, was the dominant trigger. Still seeing it after upgrading? Raise the hook `timeout` in `~/.claude/settings.json` (8s default; no CLI knob) as a secondary margin. Detail: [docs/troubleshooting.md](docs/troubleshooting.md).
|
|
178
178
|
- **Anything setup-shaped** (download blocked, server unreachable, watcher memory bloat, indexer bugs) → [docs/troubleshooting.md](docs/troubleshooting.md).
|
|
179
179
|
|
|
180
180
|
---
|
package/docs/troubleshooting.md
CHANGED
|
@@ -12,6 +12,15 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
|
|
|
12
12
|
- If both snap bun (`/snap/bin/bun`) and native bun (`~/.bun/bin/bun`) are installed, `which bun` may return the snap version. Direct `bun -e` or `bun run` commands will use the wrong binary.
|
|
13
13
|
- Fix: The `bin/clawmem` wrapper handles this automatically. For manual commands, use `~/.bun/bin/bun` explicitly or add `~/.bun/bin` to PATH before `/snap/bin`.
|
|
14
14
|
|
|
15
|
+
**macOS: `bootstrap` / `doctor` fails with "does not support dynamic extension loading" (sqlite-vec on macOS)**
|
|
16
|
+
- `clawmem bootstrap` (or `clawmem doctor`) fails at the database step because macOS's built-in SQLite — which Bun uses by default — is compiled without extension-loading support, so the `sqlite-vec` vector extension cannot load. Symptom: `✗ Database: ... This build of sqlite3 does not support dynamic extension loading`. Yoloshii/ClawMem#20.
|
|
17
|
+
- Fix: install an extension-capable SQLite via Homebrew, then re-run bootstrap:
|
|
18
|
+
```bash
|
|
19
|
+
brew install sqlite
|
|
20
|
+
clawmem bootstrap ~/notes --name notes
|
|
21
|
+
```
|
|
22
|
+
- ClawMem auto-detects Homebrew's SQLite at the standard prefixes (`/opt/homebrew` on Apple Silicon, `/usr/local` on Intel) and at `brew --prefix sqlite` for non-standard prefixes — installing it is all that's required, no env var or config. If it still fails after `brew install sqlite`, run `brew reinstall sqlite` and confirm `ls $(brew --prefix sqlite)/lib/libsqlite3.dylib` resolves.
|
|
23
|
+
|
|
15
24
|
## Embedding & GPU
|
|
16
25
|
|
|
17
26
|
**"Local model download blocked" error**
|
|
@@ -127,7 +136,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
|
|
|
127
136
|
- If the error persists after v0.1.8: restart the watcher to clear accumulated state (`systemctl --user restart clawmem-watcher.service`). Check `systemctl --user status clawmem-watcher.service` for memory usage — healthy is under 100MB, bloated is 400MB+.
|
|
128
137
|
- **v0.2.4 fix:** Hook's SQLite `busy_timeout` was 500ms — too tight. During A-MEM enrichment or heavy indexing, the watcher can hold write locks for 500ms+, causing the hook's DB open to fail with SQLITE_BUSY. Raised to 5000ms (matches MCP server). The hook's 8s outer timeout still leaves 3s for actual work after a 5s busy wait.
|
|
129
138
|
- **v0.3.1 fix:** Shell `timeout` wrappers (e.g., `timeout 8 clawmem hook context-surfacing`) kill the process with exit 124 and no stderr — Claude Code reports "Failed with non-blocking status code: No stderr output". This affects all hook events (UserPromptSubmit, Stop, SessionStart, PreCompact), not just Stop hooks. Fix: Remove shell `timeout` from all hook commands and use Claude Code's native `timeout` property instead. Run `clawmem setup hooks` to reinstall with correct config (v0.3.1+), or manually update `~/.claude/settings.json` — see [setup-hooks](guides/setup-hooks.md).
|
|
130
|
-
- **Large vault +
|
|
139
|
+
- **Large vault + intermittent hook timeout (`timed out after 8s`) — FIXED in v0.16.0.** Earlier this was diagnosed as pure cold-start (fresh Bun process, opening a large `index.sqlite`, re-reading evicted index pages) with "give the host more RAM" as the durable fix — but the dominant causes were two code-level defects: (1) the `context-surfacing` vector leg ran a *synchronous* `sqlite-vec` scan that the `Promise.race(vectorTimeout)` guard could not bound (a synchronous call blocks the event loop, so the timer never fires), and (2) every writable hook open ran an unconditional backfill `UPDATE` that could wait out `busy_timeout` under writer contention. **v0.16.0 fixes both:** `searchVec` takes a real wall-clock deadline and self-aborts before the blocking scan; both vector legs race the embed against the remaining budget and clear their timers; the init backfill is read-guarded and the init `busy_timeout` is capped to the caller's value; and the watcher prewarms the sqlite-vec payload into the page cache on startup (embed-independent, watcher-only). A cold page cache still adds latency to the genuine first post-boot call, so host RAM headroom + the prewarm help the margin — but on a large vault the scan cost, not RAM, was the trigger. A modest `timeout` bump (see the tradeoffs table under *Hooks slow or near timeout*) remains a secondary margin. The `deep` profile additionally reranks (extra remote round-trips), widening the cold-call window; `balanced` (default) does not rerank.
|
|
131
140
|
|
|
132
141
|
**Watcher memory bloat (400MB+)**
|
|
133
142
|
- The watcher accumulates memory when processing high-frequency file change events. The most common trigger was Claude Code session transcript `.jsonl` files changing on every keystroke during active conversations. Each event opened the database briefly, and over hours of active use, memory grew to 400-800MB.
|
|
@@ -213,7 +222,7 @@ Common issues when running ClawMem with hooks, MCP server, or OpenClaw plugin. O
|
|
|
213
222
|
|
|
214
223
|
The timeout applies per invocation. A slow first prompt (cold start) doesn't mean subsequent prompts will be slow — Bun caches modules after the first load, and `node-llama-cpp` model files are cached on disk after the first download. Subsequent prompts in the same session are typically faster.
|
|
215
224
|
|
|
216
|
-
**Exception —
|
|
225
|
+
**Exception — large vaults (intermittent, pre-v0.16.0):** before v0.16.0 the hook could time out on *certain* turns (not just the first) because of an unbounded synchronous `sqlite-vec` scan plus an init-time write-lock wait — see *"UserPromptSubmit hook error" (intermittent)* above. **Upgrade to v0.16.0**, which bounds the scan and the init path and prewarms the cache. Host RAM headroom + the watcher prewarm still help the genuine cold-call margin, but they were not the root cause.
|
|
217
226
|
|
|
218
227
|
**Stop hooks** (`decision-extractor`, `handoff-generator`, `feedback-loop`) default to 30s (v0.3.1+, was 10s prior) because they run LLM inference (observer model). These run at session end, so latency doesn't block the user.
|
|
219
228
|
|
package/package.json
CHANGED
package/src/clawmem.ts
CHANGED
|
@@ -8,6 +8,7 @@ import { existsSync, mkdirSync, readFileSync } from "fs";
|
|
|
8
8
|
import { resolve as pathResolve, basename } from "path";
|
|
9
9
|
import {
|
|
10
10
|
createStore,
|
|
11
|
+
prewarmVectors,
|
|
11
12
|
enableProductionMode,
|
|
12
13
|
getDefaultDbPath,
|
|
13
14
|
canonicalDocId,
|
|
@@ -1886,6 +1887,20 @@ async function cmdWatch() {
|
|
|
1886
1887
|
stopHeavyLane = startHeavyMaintenanceWorker(s, llm, cfg);
|
|
1887
1888
|
}
|
|
1888
1889
|
|
|
1890
|
+
// Prewarm the sqlite-vec chunks into OS page cache ONCE, in the single long-lived watcher
|
|
1891
|
+
// process only (never in the per-session MCP processes — N concurrent cold scans would be an
|
|
1892
|
+
// I/O storm). The context-surfacing UserPromptSubmit hook runs a SYNCHRONOUS sqlite-vec MATCH
|
|
1893
|
+
// that cannot be time-bounded in-thread (bun:sqlite exposes no interrupt); a cold ~1.5 GB scan
|
|
1894
|
+
// can blow the hook's 8-15s budget. A single warm scan here keeps the hook-path scan sub-second.
|
|
1895
|
+
// Deferred + best-effort so it never delays watcher startup and never throws.
|
|
1896
|
+
setTimeout(() => {
|
|
1897
|
+
try {
|
|
1898
|
+
// prewarmVectors is embed-independent and returns true ONLY when a scan actually ran,
|
|
1899
|
+
// so we never log a false-positive "prewarmed" (embed down at boot / no vectors yet).
|
|
1900
|
+
if (prewarmVectors(s.db)) console.log(`${c.dim}[watch] vector cache prewarmed${c.reset}`);
|
|
1901
|
+
} catch { /* best-effort: unexpected SQL error */ }
|
|
1902
|
+
}, 0);
|
|
1903
|
+
|
|
1889
1904
|
watcherHandle = startWatcher(dirs, {
|
|
1890
1905
|
debounceMs: 2000,
|
|
1891
1906
|
onChanged: async (fullPath, event) => {
|
|
@@ -183,14 +183,24 @@ export async function contextSurfacing(
|
|
|
183
183
|
// When vector succeeds, also supplement with FTS for keyword-exact recall
|
|
184
184
|
let results: SearchResult[] = [];
|
|
185
185
|
if (profile.useVector) {
|
|
186
|
+
let vectorTimer: ReturnType<typeof setTimeout> | undefined;
|
|
186
187
|
try {
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
188
|
+
// Pass a wall-clock deadline into searchVec: the Promise.race below abandons the vector
|
|
189
|
+
// promise on timeout but cannot CANCEL it (and cannot interrupt its synchronous scan). The
|
|
190
|
+
// deadline makes searchVec self-abort before the blocking MATCH, so a slow embed cannot let
|
|
191
|
+
// an already-timed-out vector leg resume and re-block the hook after it fell back to FTS.
|
|
192
|
+
const vectorDeadline = Date.now() + profile.vectorTimeout;
|
|
193
|
+
const vectorPromise = store.searchVec(retrievalQuery, DEFAULT_EMBED_MODEL, maxResults, undefined, undefined, undefined, vectorDeadline);
|
|
194
|
+
const timeoutPromise = new Promise<SearchResult[]>((_, reject) => {
|
|
195
|
+
vectorTimer = setTimeout(() => reject(new Error("vector timeout")), profile.vectorTimeout);
|
|
196
|
+
});
|
|
191
197
|
results = await Promise.race([vectorPromise, timeoutPromise]);
|
|
192
198
|
} catch {
|
|
193
199
|
// Vector search unavailable, timed out, or errored — fall back to BM25
|
|
200
|
+
} finally {
|
|
201
|
+
// Clear the timer when the vector promise won the race: a pending (ref'd) setTimeout keeps the
|
|
202
|
+
// Bun hook process alive for the full vectorTimeout after results are already in hand.
|
|
203
|
+
if (vectorTimer) clearTimeout(vectorTimer);
|
|
194
204
|
}
|
|
195
205
|
}
|
|
196
206
|
|
|
@@ -269,7 +279,21 @@ export async function contextSurfacing(
|
|
|
269
279
|
if (eq.type === 'lex') {
|
|
270
280
|
hits = store.searchFTS(eq.query, 5);
|
|
271
281
|
} else if (profile.useVector) {
|
|
272
|
-
|
|
282
|
+
// Bound BOTH the async embed wait (Promise.race on the remaining 6s budget) AND the
|
|
283
|
+
// late synchronous MATCH (the deadline arg makes the abandoned promise self-abort
|
|
284
|
+
// before the scan) — mirroring the balanced leg above. The loop guard only breaks
|
|
285
|
+
// BETWEEN iterations, so without the race a slow embed here can still blow the budget.
|
|
286
|
+
const remainingMs = startTime + 6000 - Date.now();
|
|
287
|
+
if (remainingMs <= 0) break;
|
|
288
|
+
let deepTimer: ReturnType<typeof setTimeout> | undefined;
|
|
289
|
+
try {
|
|
290
|
+
const deepVec = store.searchVec(eq.query, DEFAULT_EMBED_MODEL, 5, undefined, undefined, undefined, startTime + 6000);
|
|
291
|
+
const deepTimeout = new Promise<SearchResult[]>((_, reject) => {
|
|
292
|
+
deepTimer = setTimeout(() => reject(new Error("vector timeout")), remainingMs);
|
|
293
|
+
});
|
|
294
|
+
hits = await Promise.race([deepVec, deepTimeout]);
|
|
295
|
+
} catch { /* vector leg non-fatal (timed out or errored) */ }
|
|
296
|
+
finally { if (deepTimer) clearTimeout(deepTimer); } // don't let a pending timer keep the hook process alive
|
|
273
297
|
}
|
|
274
298
|
for (const r of hits) {
|
|
275
299
|
if (!seen.has(r.filepath)) {
|
package/src/store.ts
CHANGED
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
|
|
15
15
|
import { Database } from "bun:sqlite";
|
|
16
16
|
import { Glob } from "bun";
|
|
17
|
-
import { realpathSync } from "node:fs";
|
|
17
|
+
import { realpathSync, existsSync } from "node:fs";
|
|
18
18
|
import * as sqliteVec from "sqlite-vec";
|
|
19
19
|
import {
|
|
20
20
|
LlamaCpp,
|
|
@@ -290,17 +290,82 @@ export function toVirtualPath(db: Database, absolutePath: string): string | null
|
|
|
290
290
|
// Database initialization
|
|
291
291
|
// =============================================================================
|
|
292
292
|
|
|
293
|
-
// On macOS,
|
|
293
|
+
// On macOS, Apple's built-in libsqlite3 — which Bun uses by default — is
|
|
294
|
+
// compiled WITHOUT extension-loading support, so sqliteVec.load() fails with
|
|
295
|
+
// "This build of sqlite3 does not support dynamic extension loading" (Issue #20).
|
|
296
|
+
// Point Bun at an extension-capable SQLite (Homebrew's) via setCustomSQLite()
|
|
297
|
+
// BEFORE the first Database is opened. setCustomSQLite() must receive a path
|
|
298
|
+
// that EXISTS — an invalid path hard-crashes Bun (oven-sh/bun#18811) — so every
|
|
299
|
+
// candidate is existence-checked first. The resolved path (or null) is recorded
|
|
300
|
+
// so loadVecExtension() can emit an actionable error when no extension-capable
|
|
301
|
+
// SQLite is installed, instead of the cryptic extension-loading failure.
|
|
302
|
+
|
|
303
|
+
/** macOS only: the extension-capable SQLite activated via setCustomSQLite, or null if none was found. */
|
|
304
|
+
let macosCustomSqlitePath: string | null = null;
|
|
305
|
+
|
|
294
306
|
if (process.platform === "darwin") {
|
|
295
|
-
const
|
|
307
|
+
const candidates = [
|
|
308
|
+
"/opt/homebrew/opt/sqlite/lib/libsqlite3.dylib", // Homebrew (Apple Silicon)
|
|
309
|
+
"/usr/local/opt/sqlite/lib/libsqlite3.dylib", // Homebrew (Intel)
|
|
310
|
+
];
|
|
311
|
+
// For a non-standard Homebrew prefix, ask brew directly — but only when the
|
|
312
|
+
// standard paths are absent, so the common case pays no subprocess cost.
|
|
313
|
+
if (!candidates.some(p => existsSync(p))) {
|
|
314
|
+
try {
|
|
315
|
+
const brew = Bun.spawnSync(["brew", "--prefix", "sqlite"], { stdout: "pipe", stderr: "ignore" });
|
|
316
|
+
const prefix = brew.success ? brew.stdout.toString().trim() : "";
|
|
317
|
+
if (prefix) candidates.push(`${prefix}/lib/libsqlite3.dylib`);
|
|
318
|
+
} catch { /* brew not installed — nothing more to probe */ }
|
|
319
|
+
}
|
|
320
|
+
for (const candidate of candidates) {
|
|
321
|
+
if (!existsSync(candidate)) continue;
|
|
322
|
+
try {
|
|
323
|
+
Database.setCustomSQLite(candidate);
|
|
324
|
+
macosCustomSqlitePath = candidate;
|
|
325
|
+
break;
|
|
326
|
+
} catch { /* not usable — try the next candidate */ }
|
|
327
|
+
}
|
|
328
|
+
}
|
|
329
|
+
|
|
330
|
+
/**
|
|
331
|
+
* Translate a sqlite-vec load failure into an actionable error on macOS, where
|
|
332
|
+
* the default system SQLite cannot load extensions (Issue #20). Pure + exported
|
|
333
|
+
* for tests: pass `platform` / `foundPath` explicitly to exercise either branch.
|
|
334
|
+
* Non-macOS, or any unrelated error, is returned unchanged.
|
|
335
|
+
*/
|
|
336
|
+
export function explainVecLoadError(
|
|
337
|
+
err: unknown,
|
|
338
|
+
platform: string = process.platform,
|
|
339
|
+
foundPath: string | null = macosCustomSqlitePath,
|
|
340
|
+
): Error {
|
|
341
|
+
const original = err instanceof Error ? err : new Error(String(err));
|
|
342
|
+
const isExtensionError = /does not support dynamic extension loading/i.test(original.message);
|
|
343
|
+
if (platform !== "darwin" || !isExtensionError) return original;
|
|
344
|
+
|
|
345
|
+
const detail = foundPath === null
|
|
346
|
+
? "No Homebrew SQLite was found at the standard locations (/opt/homebrew or /usr/local)."
|
|
347
|
+
: `A custom SQLite was set from ${foundPath}, but it still cannot load extensions — try 'brew reinstall sqlite'.`;
|
|
348
|
+
return new Error(
|
|
349
|
+
"ClawMem could not load the sqlite-vec extension: macOS's built-in SQLite is " +
|
|
350
|
+
"compiled without extension support.\n" +
|
|
351
|
+
"Fix: install an extension-capable SQLite with Homebrew, then re-run:\n" +
|
|
352
|
+
" brew install sqlite\n" +
|
|
353
|
+
`${detail}\n` +
|
|
354
|
+
"More detail: docs/troubleshooting.md (\"Bun runtime\" -> sqlite-vec on macOS), Yoloshii/ClawMem#20.\n" +
|
|
355
|
+
`Original error: ${original.message}`,
|
|
356
|
+
);
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
/** Load the sqlite-vec extension, surfacing an actionable error on macOS (Issue #20). */
|
|
360
|
+
function loadVecExtension(db: Database): void {
|
|
296
361
|
try {
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
}
|
|
362
|
+
sqliteVec.load(db);
|
|
363
|
+
} catch (err) {
|
|
364
|
+
throw explainVecLoadError(err);
|
|
365
|
+
}
|
|
301
366
|
}
|
|
302
367
|
|
|
303
|
-
function initializeDatabase(db: Database): void {
|
|
368
|
+
function initializeDatabase(db: Database, busyTimeoutMs: number = 15000): void {
|
|
304
369
|
// Set busy_timeout FIRST so subsequent PRAGMAs (journal_mode in particular,
|
|
305
370
|
// which acquires a write lock when switching or initializing WAL state) wait
|
|
306
371
|
// instead of returning SQLITE_BUSY when concurrent Stop hooks
|
|
@@ -308,11 +373,13 @@ function initializeDatabase(db: Database): void {
|
|
|
308
373
|
// before_reset hook fan-out in src/openclaw/engine.ts — open the DB
|
|
309
374
|
// simultaneously. busy_timeout is a connection-level setting that only
|
|
310
375
|
// governs *subsequent* statements (default busy handler is NULL → SQLITE_BUSY
|
|
311
|
-
// returns immediately), so it must precede the contending PRAGMAs.
|
|
312
|
-
// well within the 30s Stop hook timeout
|
|
376
|
+
// returns immediately), so it must precede the contending PRAGMAs. The init
|
|
377
|
+
// busy_timeout defaults to 15s (well within the 30s Stop hook timeout) but is
|
|
378
|
+
// capped to the caller's opts.busyTimeout — hook opens pass 5000 so init cannot
|
|
379
|
+
// wait out the 8-15s UserPromptSubmit budget. createStore() resets to operational
|
|
313
380
|
// value (5000ms or opts.busyTimeout) after DDL completes. Issue #13.
|
|
314
|
-
db.exec(
|
|
315
|
-
|
|
381
|
+
db.exec(`PRAGMA busy_timeout = ${busyTimeoutMs}`);
|
|
382
|
+
loadVecExtension(db);
|
|
316
383
|
db.exec("PRAGMA journal_mode = WAL");
|
|
317
384
|
db.exec("PRAGMA foreign_keys = ON");
|
|
318
385
|
|
|
@@ -385,9 +452,17 @@ function initializeDatabase(db: Database): void {
|
|
|
385
452
|
}
|
|
386
453
|
}
|
|
387
454
|
|
|
388
|
-
// Backfill last_accessed_at from modified_at for existing docs
|
|
455
|
+
// Backfill last_accessed_at from modified_at for existing docs.
|
|
456
|
+
// Guarded by a read first: on an already-backfilled DB the UPDATE is skipped, so a writable
|
|
457
|
+
// open takes NO write lock here and cannot wait on busy_timeout under concurrent writers.
|
|
458
|
+
// (The unconditional UPDATE previously ran on EVERY writable open — including the
|
|
459
|
+
// context-surfacing UserPromptSubmit hook — and could block up to busy_timeout when another
|
|
460
|
+
// process held the write lock, pushing the hook past its 8-15s deadline.)
|
|
389
461
|
try {
|
|
390
|
-
db.
|
|
462
|
+
const needsBackfill = db.prepare(`SELECT 1 FROM documents WHERE last_accessed_at IS NULL LIMIT 1`).get();
|
|
463
|
+
if (needsBackfill) {
|
|
464
|
+
db.exec(`UPDATE documents SET last_accessed_at = modified_at WHERE last_accessed_at IS NULL`);
|
|
465
|
+
}
|
|
391
466
|
} catch { /* ignore if already backfilled */ }
|
|
392
467
|
|
|
393
468
|
db.exec(`CREATE INDEX IF NOT EXISTS idx_documents_collection ON documents(collection, active)`);
|
|
@@ -1071,6 +1146,22 @@ export function getVecTableDim(db: Database): number | null {
|
|
|
1071
1146
|
return readVecTableDim(db);
|
|
1072
1147
|
}
|
|
1073
1148
|
|
|
1149
|
+
/**
|
|
1150
|
+
* Prewarm the sqlite-vec payload into OS page cache with a single brute-force MATCH using a ZERO
|
|
1151
|
+
* query vector. Decoupled from the embedding server on purpose — it works when the embed server is
|
|
1152
|
+
* down at boot or CLAWMEM_NO_LOCAL_MODELS=true, unlike a searchVec()-based prewarm (which embeds
|
|
1153
|
+
* first and would silently no-op). Returns true ONLY if a scan actually ran (a vector table with a
|
|
1154
|
+
* known dimension exists), so callers never log a false-positive "warmed". The k-NN MATCH is
|
|
1155
|
+
* brute-force, so it touches every vector chunk — exactly the payload we want cache-resident.
|
|
1156
|
+
*/
|
|
1157
|
+
export function prewarmVectors(db: Database): boolean {
|
|
1158
|
+
const dim = getVecTableDim(db);
|
|
1159
|
+
if (!dim || dim <= 0) return false;
|
|
1160
|
+
const zero = new Float32Array(dim);
|
|
1161
|
+
db.prepare(`SELECT hash_seq FROM vectors_vec WHERE embedding MATCH ? AND k = ?`).all(zero, 1);
|
|
1162
|
+
return true;
|
|
1163
|
+
}
|
|
1164
|
+
|
|
1074
1165
|
/**
|
|
1075
1166
|
* The DISTINCT non-empty embedding models stored in the vault (empty array if no
|
|
1076
1167
|
* embeddings exist). Used to detect model drift BETWEEN runs — a different model at
|
|
@@ -1139,7 +1230,7 @@ export type Store = {
|
|
|
1139
1230
|
|
|
1140
1231
|
// Search
|
|
1141
1232
|
searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => SearchResult[];
|
|
1142
|
-
searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => Promise<SearchResult[]>;
|
|
1233
|
+
searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => Promise<SearchResult[]>;
|
|
1143
1234
|
|
|
1144
1235
|
// Query expansion & reranking
|
|
1145
1236
|
expandQuery: (query: string, model?: string, intent?: string) => Promise<ExpandedQuery[]>;
|
|
@@ -1274,7 +1365,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
|
|
|
1274
1365
|
? new Database(resolvedPath, { readonly: true })
|
|
1275
1366
|
: new Database(resolvedPath);
|
|
1276
1367
|
if (!opts?.readonly) {
|
|
1277
|
-
initializeDatabase(db);
|
|
1368
|
+
initializeDatabase(db, opts?.busyTimeout ?? 15000);
|
|
1278
1369
|
} else {
|
|
1279
1370
|
// Readonly: set busy_timeout FIRST so the journal_mode PRAGMA below
|
|
1280
1371
|
// doesn't race when concurrent processes open the DB. PRAGMA
|
|
@@ -1283,11 +1374,11 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
|
|
|
1283
1374
|
// production caller in this repo currently passes readonly:true,
|
|
1284
1375
|
// but the ordering invariant should hold regardless. Issue #13.
|
|
1285
1376
|
db.exec(`PRAGMA busy_timeout = ${opts?.busyTimeout ?? 5000}`);
|
|
1286
|
-
|
|
1377
|
+
loadVecExtension(db);
|
|
1287
1378
|
db.exec("PRAGMA journal_mode = WAL");
|
|
1288
1379
|
db.exec("PRAGMA query_only = ON");
|
|
1289
1380
|
}
|
|
1290
|
-
// For the writable branch: initializeDatabase() set 15000 during DDL —
|
|
1381
|
+
// For the writable branch: initializeDatabase() set opts.busyTimeout (default 15000) during DDL —
|
|
1291
1382
|
// reset to operational value here. For readonly: already set inside the
|
|
1292
1383
|
// branch above; this assignment is a no-op rewrite to the same value.
|
|
1293
1384
|
db.exec(`PRAGMA busy_timeout = ${opts?.busyTimeout ?? 5000}`);
|
|
@@ -1333,7 +1424,7 @@ export function createStore(dbPath?: string, opts?: { readonly?: boolean; busyTi
|
|
|
1333
1424
|
|
|
1334
1425
|
// Search
|
|
1335
1426
|
searchFTS: (query: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchFTS(db, query, limit, collectionId, collections, dateRange),
|
|
1336
|
-
searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }) => searchVec(db, query, model, limit, collectionId, collections, dateRange),
|
|
1427
|
+
searchVec: (query: string, model: string, limit?: number, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number) => searchVec(db, query, model, limit, collectionId, collections, dateRange, deadlineMs),
|
|
1337
1428
|
|
|
1338
1429
|
// Query expansion & reranking
|
|
1339
1430
|
expandQuery: (query: string, model?: string, intent?: string) => expandQuery(query, model, db, intent),
|
|
@@ -3296,13 +3387,19 @@ export function searchFTS(db: Database, query: string, limit: number = 20, colle
|
|
|
3296
3387
|
// Vector Search
|
|
3297
3388
|
// =============================================================================
|
|
3298
3389
|
|
|
3299
|
-
export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }): Promise<SearchResult[]> {
|
|
3390
|
+
export async function searchVec(db: Database, query: string, model: string, limit: number = 20, collectionId?: number, collections?: string[], dateRange?: { start: string; end: string }, deadlineMs?: number): Promise<SearchResult[]> {
|
|
3300
3391
|
const tableExists = db.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name='vectors_vec'`).get();
|
|
3301
3392
|
if (!tableExists) return [];
|
|
3302
3393
|
|
|
3303
3394
|
const embedding = await getEmbedding(query, model, true);
|
|
3304
3395
|
if (!embedding) return [];
|
|
3305
3396
|
|
|
3397
|
+
// Guard-defect fix: the caller's Promise.race(vectorTimeout) cannot interrupt the SYNCHRONOUS
|
|
3398
|
+
// sqlite-vec MATCH below (bun:sqlite blocks the event loop) and does NOT cancel this promise.
|
|
3399
|
+
// If the wall-clock budget already elapsed during the async embed above, bail here so a
|
|
3400
|
+
// timed-out vector leg cannot resume and re-block the hook after it fell back to FTS.
|
|
3401
|
+
if (deadlineMs !== undefined && Date.now() >= deadlineMs) return [];
|
|
3402
|
+
|
|
3306
3403
|
// IMPORTANT: We use a two-step query approach here because sqlite-vec virtual tables
|
|
3307
3404
|
// hang indefinitely when combined with JOINs in the same query. Do NOT try to
|
|
3308
3405
|
// "optimize" this by combining into a single query with JOINs - it will break.
|