clawmem 0.1.2 → 0.1.4

package/AGENTS.md

@@ -257,6 +257,8 @@ ClawMem hooks handle ~90% of retrieval automatically. Agent-initiated MCP calls
 
 **Hook blind spots (by design):** Hooks filter out `_clawmem/` system artifacts, enforce score thresholds, and cap token budget. Absence in `<vault-context>` does NOT mean absence in memory. If you expect a memory to exist but it wasn't surfaced, escalate to Tier 3.
 
+**Adaptive thresholds:** Context-surfacing uses ratio-based scoring that adapts to vault characteristics (size, document quality, content age, embedding model). Results are kept within a percentage of the best result's composite score rather than a fixed absolute threshold. An activation floor prevents surfacing when all results are weak. Profiles control the ratio: `speed` (65%), `balanced` (55%), `deep` (45% + query expansion + reranking). `CLAWMEM_PROFILE=deep` is recommended for vaults with older content or lower-quality documents. MCP tools use fixed absolute thresholds, not adaptive.
+
 ### Tier 3 — Agent-Initiated (one targeted MCP call)
 
 **Escalate ONLY when one of these three rules fires:**

package/CLAUDE.md

@@ -257,6 +257,8 @@ ClawMem hooks handle ~90% of retrieval automatically. Agent-initiated MCP calls
 
 **Hook blind spots (by design):** Hooks filter out `_clawmem/` system artifacts, enforce score thresholds, and cap token budget. Absence in `<vault-context>` does NOT mean absence in memory. If you expect a memory to exist but it wasn't surfaced, escalate to Tier 3.
 
+**Adaptive thresholds:** Context-surfacing uses ratio-based scoring that adapts to vault characteristics (size, document quality, content age, embedding model). Results are kept within a percentage of the best result's composite score rather than a fixed absolute threshold. An activation floor prevents surfacing when all results are weak. Profiles control the ratio: `speed` (65%), `balanced` (55%), `deep` (45% + query expansion + reranking). `CLAWMEM_PROFILE=deep` is recommended for vaults with older content or lower-quality documents. MCP tools use fixed absolute thresholds, not adaptive.
+
 ### Tier 3 — Agent-Initiated (one targeted MCP call)
 
 **Escalate ONLY when one of these three rules fires:**

package/README.md

@@ -63,7 +63,7 @@ Runs fully local with no API keys and no cloud services. Integrates via Claude C
 
 **Required:**
 
-- [Bun](https://bun.sh) v1.0+ — runtime for ClawMem
+- [Bun](https://bun.sh) v1.0+ — runtime for ClawMem. On Linux, install via `curl -fsSL https://bun.sh/install | bash` (not snap — snap Bun cannot read stdin, which breaks hooks).
 - SQLite with FTS5 — included with Bun. On macOS, install `brew install sqlite` for extension loading support (ClawMem detects and uses Homebrew SQLite automatically).
 
 **Optional (for better performance):**
@@ -1015,6 +1015,21 @@ Built on the shoulders of:
 - [SAME](https://github.com/sgx-labs/statelessagent) — agent memory concepts (recency decay, confidence scoring, session tracking)
 - [supermemory](https://github.com/supermemoryai/clawdbot-supermemory) — hook patterns and context surfacing ideas
 
+## Roadmap
+
+| Status | Feature | Description |
+|--------|---------|-------------|
+| :white_check_mark: | Adaptive thresholds | Ratio-based filtering that adapts to vault characteristics (v0.1.3) |
+| :white_check_mark: | Deep escalation | Budget-aware query expansion + cross-encoder reranking for `deep` profile |
+| :white_check_mark: | Cloud embedding providers | Jina, OpenAI, Voyage, Cohere with batch embedding + TPM pacing |
+| :construction: | Calibration probes | One-time `clawmem calibrate` command that measures your vault's score distribution and tunes thresholds automatically |
+| :construction: | Rolling threshold learning | Learns optimal activation floor from actual usage patterns (which surfaced content gets referenced vs ignored) |
+| :memo: | Multi-vault namespacing | Isolated vaults with independent calibration and lifecycle policies |
+| :memo: | REST API authentication | Token-scoped access for multi-agent deployments |
+| :memo: | Streaming rerank | Cross-encoder reranking within the hook timeout budget for larger candidate sets |
+
+:white_check_mark: Shipped  :construction: Planned  :memo: Exploring
+
 ## License
 
 MIT

package/SKILL.md

@@ -191,6 +191,8 @@ Hooks handle ~90% of retrieval. Zero agent effort.
 
 **Hook blind spots (by design):** Hooks filter out `_clawmem/` system artifacts, enforce score thresholds, and cap token budget. Absence in `<vault-context>` does NOT mean absence in memory. Escalate to Tier 3 if expected memory wasn't surfaced.
 
+**Adaptive thresholds:** Context-surfacing uses ratio-based scoring that adapts to vault characteristics. Results are kept within a percentage of the best result's composite score (speed: 65%, balanced: 55%, deep: 45%). An activation floor prevents surfacing when all results are weak. `CLAWMEM_PROFILE=deep` adds query expansion + reranking. MCP tools use fixed absolute thresholds, not adaptive.
+
 ---
 
 ## Tier 3 — Agent-Initiated Retrieval (MCP Tools)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clawmem",
-  "version": "0.1.2",
+  "version": "0.1.4",
   "description": "On-device context engine and memory for AI agents. Claude Code and OpenClaw. Hooks + MCP server + hybrid RAG search.",
   "type": "module",
   "bin": {

package/src/config.ts

@@ -70,13 +70,26 @@ export interface ProfileConfig {
   maxResults: number;
   useVector: boolean;
   vectorTimeout: number;
+  /** Legacy absolute threshold — used by MCP tools and as fallback when thresholdMode="absolute" */
   minScore: number;
+  /** Adaptive: keep results within this ratio of best score (e.g., 0.55 = top 55%) */
+  minScoreRatio: number;
+  /** Adaptive: never surface below this regardless of ratio */
+  absoluteFloor: number;
+  /** Adaptive: if best result is below this, return empty (prevents all-weak surfacing) */
+  activationFloor: number;
+  /** "adaptive" uses ratio-based filtering; "absolute" uses legacy minScore */
+  thresholdMode: "adaptive" | "absolute";
+  /** Budget-aware escalation: if fast path finishes early, spend remaining time on expansion + reranking */
+  deepEscalation: boolean;
+  /** Max time (ms) allowed for the fast path before escalation is considered */
+  escalationBudgetMs: number;
 }
 
 export const PROFILES: Record<PerformanceProfile, ProfileConfig> = {
-  speed:    { tokenBudget: 400,  maxResults: 5,  useVector: false, vectorTimeout: 0,    minScore: 0.55 },
-  balanced: { tokenBudget: 800,  maxResults: 10, useVector: true,  vectorTimeout: 900,  minScore: 0.45 },
-  deep:     { tokenBudget: 1200, maxResults: 15, useVector: true,  vectorTimeout: 2000, minScore: 0.35 },
+  speed:    { tokenBudget: 400,  maxResults: 5,  useVector: false, vectorTimeout: 0,    minScore: 0.55, minScoreRatio: 0.65, absoluteFloor: 0.18, activationFloor: 0.24, thresholdMode: "adaptive", deepEscalation: false, escalationBudgetMs: 0 },
+  balanced: { tokenBudget: 800,  maxResults: 10, useVector: true,  vectorTimeout: 900,  minScore: 0.45, minScoreRatio: 0.55, absoluteFloor: 0.15, activationFloor: 0.20, thresholdMode: "adaptive", deepEscalation: false, escalationBudgetMs: 0 },
+  deep:     { tokenBudget: 1200, maxResults: 15, useVector: true,  vectorTimeout: 2000, minScore: 0.25, minScoreRatio: 0.45, absoluteFloor: 0.12, activationFloor: 0.16, thresholdMode: "adaptive", deepEscalation: true,  escalationBudgetMs: 4000 },
 };
 
 export function getActiveProfile(): ProfileConfig {

package/src/hooks/context-surfacing.ts

@@ -7,7 +7,8 @@
  */
 
 import type { Store, SearchResult } from "../store.ts";
-import { DEFAULT_EMBED_MODEL, extractSnippet } from "../store.ts";
+import { DEFAULT_EMBED_MODEL, DEFAULT_QUERY_MODEL, DEFAULT_RERANK_MODEL, extractSnippet, resolveStore } from "../store.ts";
+import { getVaultPath, getActiveProfile } from "../config.ts";
 import type { HookInput, HookOutput } from "../hooks.ts";
 import {
   makeContextOutput,
@@ -29,13 +30,12 @@ import { enrichResults } from "../search-utils.ts";
 import { sanitizeSnippet } from "../promptguard.ts";
 import { shouldSkipRetrieval, isRetrievedNoise } from "../retrieval-gate.ts";
 import { MAX_QUERY_LENGTH } from "../limits.ts";
-import { getActiveProfile } from "../config.ts";
 
 // =============================================================================
 // Config
 // =============================================================================
 
-// Profile-driven defaults (overridden by CLAWMEM_PROFILE env var)
+// Profile-driven defaults (overridden by CLAWMEM_PROFILE env var via E14)
 const DEFAULT_TOKEN_BUDGET = 800;
 const DEFAULT_MAX_RESULTS = 10;
 const DEFAULT_MIN_SCORE = 0.45;
@@ -52,7 +52,7 @@ function getTierConfig(score: number): { snippetLen: number; showMeta: boolean;
 // Directories to never surface
 const FILTERED_PATHS = ["_PRIVATE/", "experiments/", "_clawmem/"];
 
-// File path patterns to extract from prompts (E13: file-aware UserPromptSubmit)
+// File path patterns to extract from prompts (E13 replacement: file-aware UserPromptSubmit)
 const FILE_PATH_RE = /(?:^|\s)((?:\/[\w.@-]+)+(?:\.\w+)?|[\w.@-]+\.(?:ts|js|py|md|sh|yaml|yml|json|toml|rs|go|tsx|jsx|css|html))\b/g;
 
 // =============================================================================
@@ -81,10 +81,11 @@ export async function contextSurfacing(
     return makeEmptyOutput("context-surfacing");
   }
 
-  // Load active performance profile
+  // Load active performance profile (E14)
   const profile = getActiveProfile();
   const maxResults = profile.maxResults;
   const tokenBudget = profile.tokenBudget;
+  const startTime = Date.now();
 
   const isRecency = hasRecencyIntent(prompt);
   const minScore = isRecency ? MIN_COMPOSITE_SCORE_RECENCY : profile.minScore;
@@ -118,7 +119,22 @@ export async function contextSurfacing(
     }
   }
 
-  // File-aware supplemental search (E13): extract file paths/names from prompt
+  // Dual-query: also search skill vault if configured (secondary source)
+  if (getVaultPath("skill")) {
+    try {
+      const skillStore = resolveStore("skill");
+      const skillResults = skillStore.searchFTS(prompt, 5);
+      // Tag skill vault results for identification in output
+      for (const r of skillResults) {
+        (r as any)._fromVault = "skill";
+      }
+      results = [...results, ...skillResults];
+    } catch {
+      // Skill vault unavailable — continue with general results only
+    }
+  }
+
+  // File-aware supplemental search (E13 replacement): extract file paths/names from prompt
   // and run targeted FTS queries to surface file-specific vault context
   const fileMatches = [...prompt.matchAll(FILE_PATH_RE)].map(m => m[1]!.trim()).filter(Boolean);
   if (fileMatches.length > 0) {
@@ -138,6 +154,57 @@ export async function contextSurfacing(
 
   if (results.length === 0) return makeEmptyOutput("context-surfacing");
 
+  // Budget-aware deep escalation (deep profile only):
+  // If the fast path finished quickly and found results, spend remaining time budget
+  // on query expansion (discovers new candidates) and cross-encoder reranking (reorders).
+  if (profile.deepEscalation && results.length >= 2) {
+    const elapsed = Date.now() - startTime;
+    if (elapsed < profile.escalationBudgetMs) {
+      try {
+        // Phase 1: Query expansion — discover candidates BM25+vector missed
+        const expanded = await store.expandQuery(prompt, DEFAULT_QUERY_MODEL);
+        if (expanded.length > 0) {
+          const seen = new Set(results.map(r => r.filepath));
+          for (const eq of expanded.slice(0, 3)) {
+            if (Date.now() - startTime > 6000) break; // hard stop at 6s
+            const ftsExp = store.searchFTS(eq, 5);
+            for (const r of ftsExp) {
+              if (!seen.has(r.filepath)) {
+                seen.add(r.filepath);
+                results.push(r);
+              }
+            }
+          }
+        }
+
+        // Phase 2: Cross-encoder reranking — reorder with deeper relevance signal
+        // Sort by score first so reranking covers the best candidates, not just
+        // the first-inserted (expansion hits appended later would otherwise be missed)
+        if (Date.now() - startTime < 6000 && results.length >= 3) {
+          results.sort((a, b) => b.score - a.score);
+          const toRerank = results.slice(0, 15).map(r => ({
+            file: r.filepath,
+            text: (r.body || "").slice(0, 2000),
+          }));
+          const reranked = await store.rerank(prompt, toRerank, DEFAULT_RERANK_MODEL);
+          if (reranked.length > 0) {
+            const rerankedMap = new Map(reranked.map(r => [r.file, r.score]));
+            // Blend: 60% original score + 40% reranker score for stability
+            for (const r of results) {
+              const rerankScore = rerankedMap.get(r.filepath);
+              if (rerankScore !== undefined) {
+                r.score = 0.6 * r.score + 0.4 * rerankScore;
+              }
+            }
+            results.sort((a, b) => b.score - a.score);
+          }
+        }
+      } catch {
+        // Escalation failed (GPU down, timeout, etc.) — continue with fast-path results
+      }
+    }
+  }
+
   // Filter out private/excluded paths
   results = results.filter(r =>
     !FILTERED_PATHS.some(p => r.displayPath.includes(p))
@@ -148,8 +215,12 @@ export async function contextSurfacing(
   // Filter out snoozed documents
   const now = new Date();
   results = results.filter(r => {
+    // filepath is a virtual path (clawmem://collection/path) but findActiveDocument
+    // expects the collection-relative path, not the full virtual path
     const parsed = r.filepath.startsWith('clawmem://') ? r.filepath.replace(/^clawmem:\/\/[^/]+\/?/, '') : r.filepath;
-    const doc = store.findActiveDocument(r.collectionName, parsed);
+    // Use the correct store for skill-vault results
+    const targetStore = (r as any)._fromVault === "skill" ? (() => { try { return resolveStore("skill"); } catch { return store; } })() : store;
+    const doc = targetStore.findActiveDocument(r.collectionName, parsed);
     if (!doc) return true;
     if (doc.snoozed_until && new Date(doc.snoozed_until) > now) return false;
     return true;
@@ -170,12 +241,41 @@ export async function contextSurfacing(
   // Filter out noise results (agent denials, too-short snippets) before enrichment
   results = results.filter(r => !r.body || !isRetrievedNoise(r.body));
 
-  // Enrich with SAME metadata
-  const enriched = enrichResults(store, results, prompt);
+  // Enrich with SAME metadata — route skill-vault results through their own store
+  const generalResults = results.filter(r => !(r as any)._fromVault);
+  const skillResults = results.filter(r => (r as any)._fromVault === "skill");
+  let enriched = enrichResults(store, generalResults, prompt);
+  if (skillResults.length > 0) {
+    try {
+      const skillStore = resolveStore("skill");
+      enriched = [...enriched, ...enrichResults(skillStore, skillResults, prompt)];
+    } catch {
+      // Skill store unavailable — enrich with general store as fallback
+      enriched = [...enriched, ...enrichResults(store, skillResults, prompt)];
+    }
+  }
 
   // Apply composite scoring
-  const scored = applyCompositeScoring(enriched, prompt)
-    .filter(r => r.compositeScore >= minScore);
+  const allScored = applyCompositeScoring(enriched, prompt);
+
+  // Threshold filtering — adaptive (ratio-based) or absolute (legacy)
+  let scored: typeof allScored;
+  if (profile.thresholdMode === "adaptive") {
+    // Use max composite score across the set (not positional [0], which may be
+    // reordered by recency-intent sorting in applyCompositeScoring)
+    const bestScore = allScored.length > 0
+      ? Math.max(...allScored.map(r => r.compositeScore))
+      : 0;
+
+    // Activation floor: if even the best result is too weak, bail entirely
+    if (bestScore < profile.activationFloor) return makeEmptyOutput("context-surfacing");
+
+    const adaptiveMin = Math.max(bestScore * profile.minScoreRatio, profile.absoluteFloor);
+    scored = allScored.filter(r => r.compositeScore >= adaptiveMin);
+  } else {
+    // Legacy absolute threshold (backward compat)
+    scored = allScored.filter(r => r.compositeScore >= minScore);
+  }
 
   if (scored.length === 0) return makeEmptyOutput("context-surfacing");
 
@@ -191,6 +291,7 @@ export async function contextSurfacing(
         for (const ca of coActs) {
           const existing = scored.find(r => r.displayPath === ca.path);
           if (existing && existing.compositeScore <= 0.8) {
+            // Boost by 0.1 per co-activation count, capped at +0.2
             existing.compositeScore += Math.min(0.2, 0.1 * Math.min(ca.count, 2));
           }
         }
@@ -202,12 +303,14 @@ export async function contextSurfacing(
   }
 
   // Memory type diversification (E10): ensure procedural results aren't crowded out
+  // If top results are all semantic, promote the best procedural result
   if (scored.length > 3) {
     const top3Types = scored.slice(0, 3).map(r => inferMemoryType(r.displayPath, r.contentType, r.body));
     const hasProc = top3Types.includes("procedural");
     if (!hasProc) {
       const procIdx = scored.findIndex(r => inferMemoryType(r.displayPath, r.contentType, r.body) === "procedural");
       if (procIdx > 3) {
+        // Move the best procedural result to position 3
         const [proc] = scored.splice(procIdx, 1);
         scored.splice(3, 0, proc!);
       }
@@ -225,6 +328,7 @@ export async function contextSurfacing(
   }
 
   // Routing hint: detect query intent signals and prepend a tool routing directive
+  // This makes routing instructions salient at the moment of tool selection (per research)
   const routingHint = detectRoutingHint(prompt);
 
   return makeContextOutput(
@@ -247,14 +351,17 @@ export async function contextSurfacing(
 function detectRoutingHint(prompt: string): string | null {
   const q = prompt.toLowerCase();
 
+  // Timeline/session signals
   if (/\b(last session|yesterday|prior session|previous session|last time we|handoff|what happened last|what did we do|cross.session|earlier today|what we discussed|when we last)\b/i.test(q)) {
     return "If searching memory for this: use session_log or memory_retrieve, NOT query.";
   }
 
+  // Causal signals
   if (/\b(why did|why was|why were|what caused|what led to|reason for|decided to|decision about|trade.?off|instead of|chose to)\b/i.test(q) || /^why\b/i.test(q)) {
     return "If searching memory for this: use intent_search or memory_retrieve, NOT query.";
   }
 
+  // Discovery signals
   if (/\b(similar to|related to|what else|what other|reminds? me of|like this)\b/i.test(q)) {
     return "If searching memory for this: use find_similar or memory_retrieve, NOT query.";
   }

package/src/hooks/curator-nudge.ts

@@ -9,6 +9,7 @@
 import { resolve as pathResolve } from "path";
 import { existsSync, readFileSync } from "fs";
 import type { Store } from "../store.ts";
+import { getIndexHealth } from "../store.ts";
 import type { HookInput, HookOutput } from "../hooks.ts";
 import {
   makeContextOutput,
@@ -64,11 +65,23 @@ export async function curatorNudge(
     return makeEmptyOutput("curator-nudge");
   }
 
+  // Override embedding backlog with live data (report value goes stale after embed timer runs)
+  let actions = [...report.actions];
+  try {
+    const health = getIndexHealth(store.db);
+    actions = actions.filter(a => !/documents? need embedding/i.test(a));
+    if (health.needsEmbedding > 0) {
+      actions.unshift(`${health.needsEmbedding} documents need embedding`);
+    }
+  } catch { /* fail-open: use report actions as-is */ }
+
+  if (actions.length === 0) return makeEmptyOutput("curator-nudge");
+
   // Build compact action summary within budget
   const lines = [`**Curator (${report.timestamp.slice(0, 10)}):**`];
   let tokens = estimateTokens(lines[0]!);
 
-  for (const action of report.actions) {
+  for (const action of actions) {
     const line = `- ${action}`;
     const lineTokens = estimateTokens(line);
     if (tokens + lineTokens > MAX_TOKEN_BUDGET && lines.length > 1) break;