@betterdb/memory 0.1.2 → 0.4.0

package/README.md

@@ -16,14 +16,31 @@ relevant history at the start of each new session.
 ### Install
 
 ```bash
+# 1. Copy .env.example and fill in your settings
+cp .env.example .env
+
+# 2. Install
 bunx @betterdb/memory install
 ```
 
-This will:
+The install will:
 1. Compile native hook binaries to `~/.betterdb/bin/`
 2. Register 4 lifecycle hooks with Claude Code
 3. Register the MCP server for mid-conversation tools
 4. Create the Valkey search index
+5. Save your `.env` values to `~/.betterdb/memory.json` for runtime use
+
+### Don't have Valkey?
+
+The setup skill will offer to spin one up in Docker for you. Or run it manually:
+
+```bash
+# Via CLI
+bunx @betterdb/memory docker-valkey
+
+# Or directly with Docker
+docker run -d --name betterdb-valkey -p 6379:6379 -v betterdb-valkey-data:/data valkey/valkey-search:8 valkey-server --save 60 1
+```
 
 ### How It Works
 
@@ -37,7 +54,7 @@ This will:
 ### MCP Tools
 
 Claude can use these mid-conversation:
-- `search_context` — Semantic search over past sessions
+- `search_context` — Semantic search over past sessions. Escalates project+branch → project → cross-project, and takes an optional `tags` filter (`decision`, `pattern`, `problem`, `open-thread`)
 - `store_insight` — Save a decision, pattern, or warning
 - `list_open_threads` — Show unresolved items
 - `forget` — Delete a specific memory
@@ -45,24 +62,98 @@ Claude can use these mid-conversation:
 ### CLI Commands
 
 ```bash
-bunx @betterdb/memory install    # Set up hooks + MCP server
-bunx @betterdb/memory status     # Check health
-bunx @betterdb/memory uninstall  # Remove everything
-bunx @betterdb/memory maintain   # Run aging/compression manually
+bunx @betterdb/memory install        # Set up hooks + MCP server
+bunx @betterdb/memory status         # Check health + recall scoring config
+bunx @betterdb/memory uninstall      # Remove everything
+bunx @betterdb/memory maintain       # Run aging/compression manually
+bunx @betterdb/memory forget         # Bulk-delete by scope (dry run; --apply to delete)
+                                     #   --project <name> | --all-projects --branch <b> --tags <a,b>
+bunx @betterdb/memory docker-valkey  # Manage Docker Valkey container
 ```
 
 ### Configuration
 
-Via environment variables or `~/.betterdb/memory.json`:
+Copy `.env.example` to `.env` and fill in your values before running `bunx @betterdb/memory install`. They get saved to `~/.betterdb/memory.json` and used by the compiled binaries at runtime.
+
+#### Core
 
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `BETTERDB_VALKEY_URL` | `redis://localhost:6379` | Valkey connection URL |
-| `BETTERDB_EMBED_MODEL` | auto-detect | Embedding provider |
-| `BETTERDB_SUMMARIZE_MODEL` | auto-detect | Summarization provider |
+| `BETTERDB_VALKEY_INDEX_NAME` | `betterdb-memory-index` | Valkey search index name |
 | `BETTERDB_EMBED_DIM` | `1024` | Embedding dimensions |
-| `BETTERDB_MAX_CONTEXT_MEMORIES` | `5` | Memories injected per session |
+| `BETTERDB_MAX_CONTEXT_MEMORIES` | `5` | Max memories injected per session (after gating) |
 | `BETTERDB_CONTEXT_FILE` | `.betterdb_context.md` | Context injection file |
+| `BETTERDB_ALLOW_REMOTE_FALLBACK` | `true` | Fall back to remote APIs if local models unavailable |
+
+#### Recall Gating
+
+Recall over-fetches a candidate pool, gates it by relevance, and escalates on a
+miss (project+branch → project → cross-project). Memories are stored with their
+git branch as a native thread scope and content-type tags, so recall can narrow
+to the current branch first and filter by type. `search_context` returns nothing
+only when nothing clears the bar — so a miss is honest, not a silent drop.
+
+The gate is **relative**, not an absolute similarity threshold: embed models
+compress cosine similarity into different, narrow bands (mxbai-embed-large packs
+everything into ~0.7–0.88), so a fixed threshold doesn't transfer across models.
+Instead, `floor` drops genuine noise, and hits within `margin` of the top match
+are kept; confidence comes from the scale-independent top-vs-next gap.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BETTERDB_RECALL_FLOOR` | `0.5` | Similarity floor — drops noise and loosens the store's own distance gate |
+| `BETTERDB_RECALL_MARGIN` | `0.05` | Keep hits within this similarity of the top match |
+| `BETTERDB_RECALL_SEPARATION` | `0.04` | Top-vs-next gap above which a match is "high" confidence |
+| `BETTERDB_RECALL_POOL_K` | `10` | Rung-1 over-fetch pool (project) |
+| `BETTERDB_RECALL_POOL_K_WIDE` | `20` | Rung-2/3 over-fetch pool (wider / cross-project) |
+| `BETTERDB_ALLOW_CROSS_PROJECT` | `true` | Allow escalation / `scope="all"` to search across projects |
+
+Ranking within the gated pool uses a composite score (similarity + recency +
+importance), owned by `@betterdb/agent-memory`. Recency is the system's single
+time-decay — a half-life applied at query time, not a stored per-memory aging
+pass. These knobs tune it; defaults match the store's.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BETTERDB_RECALL_HALF_LIFE_DAYS` | `7` | Age at which a memory's recency term halves |
+| `BETTERDB_RECALL_WEIGHT_SIMILARITY` | `0.6` | Weight of semantic similarity in the composite score |
+| `BETTERDB_RECALL_WEIGHT_RECENCY` | `0.25` | Weight of recency |
+| `BETTERDB_RECALL_WEIGHT_IMPORTANCE` | `0.15` | Weight of stored importance |
+
+#### Model Providers
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BETTERDB_EMBED_PROVIDER` | auto-detect | Force embed provider: `local`, `ollama`, `voyage`, `openai`, `groq`, `together` |
+| `BETTERDB_SUMMARIZE_PROVIDER` | auto-detect | Force summarize provider: `ollama`, `anthropic`, `openai`, `groq`, `together` |
+| `BETTERDB_EMBED_MODEL` | `mxbai-embed-large` | Ollama embedding model name |
+| `BETTERDB_SUMMARIZE_MODEL` | `mistral:7b` | Ollama summarization model name |
+| `BETTERDB_OLLAMA_URL` | `http://localhost:11434` | Ollama API URL |
+
+#### Embeddings work with zero config
+
+If no embedding provider is detected (no Ollama models, no API keys), BetterDB falls back to **on-device embeddings** via `@xenova/transformers` (`all-MiniLM-L6-v2`, 384-dim, Apache-2.0). No API key, no running service — the model weights download once on first use and are cached thereafter. Auto-detected providers (Ollama, then API keys) take priority when available.
+
+#### API Keys
+
+Embeddings always work (on-device fallback above). A summarization provider is still required — Ollama is free and local; the others require API keys.
+
+| Variable | Provider | Used for |
+|----------|----------|----------|
+| `ANTHROPIC_API_KEY` | [Anthropic](https://console.anthropic.com/) | Summarization only (no embeddings) |
+| `VOYAGE_API_KEY` | [Voyage AI](https://www.voyageai.com/) | Embeddings only |
+| `OPENAI_API_KEY` | [OpenAI](https://platform.openai.com/) | Embeddings + summarization |
+| `GROQ_API_KEY` | [Groq](https://console.groq.com/) | Embeddings + summarization |
+| `TOGETHER_API_KEY` | [Together AI](https://www.together.ai/) | Embeddings + summarization |
+
+#### Aging Pipeline
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BETTERDB_COMPRESS_THRESHOLD` | `0.3` | Importance threshold for compression |
+| `BETTERDB_DISTILL_MIN_SESSIONS` | `5` | Min sessions before knowledge distillation |
+| `BETTERDB_AGING_INTERVAL_HOURS` | `6` | Hours between automatic aging runs |
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@betterdb/memory",
-  "version": "0.1.2",
+  "version": "0.4.0",
   "description": "BetterDB Memory for Claude Code — Valkey-powered persistent memory across sessions",
   "license": "MIT",
   "author": "BetterDB Inc. <hello@betterdb.com>",
@@ -46,9 +46,11 @@
     "typecheck": "tsc --noEmit"
   },
   "dependencies": {
+    "@betterdb/agent-memory": "^0.2.1",
     "iovalkey": "^0.2.1",
     "ollama": "^0.5.14",
     "@modelcontextprotocol/sdk": "^1.12.1",
+    "@xenova/transformers": "^2.17.2",
     "zod": "^3.24.4",
     "zod-to-json-schema": "^3.24.5",
     "@anthropic-ai/sdk": "latest"

package/scripts/aging-worker.ts

@@ -7,16 +7,19 @@
  *   bun run scripts/aging-worker.ts
  */
 import { getValkeyClient } from "../src/client/valkey.js";
+import { getPluginMemoryStore } from "../src/client/memory-store.js";
 import { createModelClient } from "../src/client/model.js";
 import { AgingPipeline } from "../src/memory/aging.js";
 
 try {
   const valkeyClient = await getValkeyClient();
   const modelClient = await createModelClient();
+  const store = await getPluginMemoryStore((t) => modelClient.embed(t));
 
-  const pipeline = new AgingPipeline(valkeyClient, modelClient);
+  const pipeline = new AgingPipeline(valkeyClient, store, modelClient);
   await pipeline.runFullPipeline();
 
+  await store.close();
   await valkeyClient.quit();
 } catch (err) {
   console.error("[betterdb] Aging worker failed:", err);

package/scripts/docker-valkey.sh

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+CONTAINER_NAME="betterdb-valkey"
+VOLUME_NAME="betterdb-valkey-data"
+PORT="${1:-6379}"
+ACTION="${2:-start}"
+
+# Check if Docker is available
+if ! command -v docker &>/dev/null; then
+  echo "ERROR: Docker is not installed."
+  echo "Install Docker: https://docs.docker.com/get-docker/"
+  exit 1
+fi
+
+case "$ACTION" in
+  start)
+    # Check if container already exists
+    if docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+      if docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+        echo "Valkey container '${CONTAINER_NAME}' is already running on port ${PORT}."
+        exit 0
+      else
+        echo "Starting existing Valkey container '${CONTAINER_NAME}'..."
+        docker start "${CONTAINER_NAME}"
+        exit 0
+      fi
+    fi
+
+    # Try valkey-search image first (includes Search module)
+    echo "Pulling valkey/valkey-search:8..."
+    if docker pull valkey/valkey-search:8 2>/dev/null; then
+      IMAGE="valkey/valkey-search:8"
+    else
+      echo "valkey-search image not available, falling back to valkey/valkey:8-alpine..."
+      docker pull valkey/valkey:8-alpine
+      IMAGE="valkey/valkey:8-alpine"
+    fi
+
+    echo "Starting Valkey container on port ${PORT}..."
+    if ! docker run -d \
+      --name "${CONTAINER_NAME}" \
+      -p "${PORT}:6379" \
+      -v "${VOLUME_NAME}:/data" \
+      "${IMAGE}" \
+      valkey-server --save 60 1 2>/dev/null; then
+      # Port likely in use — retry on 16379
+      PORT=16379
+      echo "Port conflict. Retrying on port ${PORT}..."
+      docker run -d \
+        --name "${CONTAINER_NAME}" \
+        -p "${PORT}:6379" \
+        -v "${VOLUME_NAME}:/data" \
+        "${IMAGE}" \
+        valkey-server --save 60 1
+    fi
+
+    # Wait for startup
+    sleep 2
+
+    # Verify
+    if docker exec "${CONTAINER_NAME}" valkey-cli ping | grep -q PONG; then
+      echo "Valkey is running on redis://localhost:${PORT}"
+    else
+      echo "WARNING: Container started but ping failed. Check: docker logs ${CONTAINER_NAME}"
+      exit 1
+    fi
+    ;;
+
+  stop)
+    if docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+      docker stop "${CONTAINER_NAME}"
+      echo "Valkey container stopped."
+    else
+      echo "Valkey container is not running."
+    fi
+    ;;
+
+  status)
+    if docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+      RUNNING_PORT=$(docker port "${CONTAINER_NAME}" 6379 2>/dev/null | head -1 | cut -d: -f2)
+      echo "Valkey container '${CONTAINER_NAME}' is running on port ${RUNNING_PORT:-unknown}."
+    elif docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
+      echo "Valkey container '${CONTAINER_NAME}' exists but is stopped."
+    else
+      echo "No Valkey container found."
+    fi
+    ;;
+
+  remove)
+    docker stop "${CONTAINER_NAME}" 2>/dev/null || true
+    docker rm "${CONTAINER_NAME}" 2>/dev/null || true
+    echo "Valkey container removed. Volume '${VOLUME_NAME}' preserved."
+    echo "To remove data: docker volume rm ${VOLUME_NAME}"
+    ;;
+
+  *)
+    echo "Usage: docker-valkey.sh [port] [start|stop|status|remove]"
+    exit 1
+    ;;
+esac

package/scripts/register-hooks.ts

@@ -0,0 +1,94 @@
+#!/usr/bin/env bun
+
+/**
+ * Register BetterDB Memory lifecycle hooks in ~/.claude/settings.json.
+ *
+ * Usage:
+ *   bun run scripts/register-hooks.ts <plugin-root>
+ *
+ * <plugin-root> is the absolute path to the plugin directory (where src/hooks/ lives).
+ * Hook commands are written with resolved absolute paths — no env vars.
+ */
+
+import { existsSync, readFileSync, writeFileSync, mkdirSync } from "node:fs";
+import { join, resolve } from "node:path";
+
+const pluginRoot = process.argv[2];
+if (!pluginRoot) {
+  console.error("Usage: bun run register-hooks.ts <plugin-root>");
+  process.exit(1);
+}
+
+const resolvedRoot = resolve(pluginRoot);
+const hooksDir = join(resolvedRoot, "src", "hooks");
+
+// Verify hook source files exist
+const hookFiles = ["session-start.ts", "pre-tool.ts", "post-tool.ts", "session-end.ts"];
+for (const file of hookFiles) {
+  if (!existsSync(join(hooksDir, file))) {
+    console.error(`ERROR: Hook source not found: ${join(hooksDir, file)}`);
+    process.exit(1);
+  }
+}
+
+const HOME = process.env["HOME"] ?? process.env["USERPROFILE"] ?? "";
+const claudeDir = join(HOME, ".claude");
+const settingsPath = join(claudeDir, "settings.json");
+
+// Ensure ~/.claude/ exists
+mkdirSync(claudeDir, { recursive: true });
+
+// Read existing settings (or start fresh)
+let settings: Record<string, unknown> = {};
+if (existsSync(settingsPath)) {
+  try {
+    settings = JSON.parse(readFileSync(settingsPath, "utf-8"));
+  } catch {
+    // Corrupted file — start fresh but warn
+    console.warn("WARNING: Could not parse ~/.claude/settings.json — existing content will be preserved as backup.");
+    const backupPath = settingsPath + ".bak";
+    writeFileSync(backupPath, readFileSync(settingsPath));
+    console.warn(`  Backup saved to ${backupPath}`);
+  }
+}
+
+function cmd(hookFile: string): string {
+  return `bash -c 'bun run "${join(hooksDir, hookFile)}"'`;
+}
+
+// Merge hooks — replaces BetterDB entries per event, preserves all others
+const existingHooks = (settings["hooks"] ?? {}) as Record<string, unknown[]>;
+const betterdbHooks: Record<string, unknown[]> = {
+  SessionStart: [
+    { hooks: [{ type: "command", command: cmd("session-start.ts") }] },
+  ],
+  PreToolUse: [
+    { matcher: "", hooks: [{ type: "command", command: cmd("pre-tool.ts") }] },
+  ],
+  PostToolUse: [
+    { matcher: "", hooks: [{ type: "command", command: cmd("post-tool.ts") }] },
+  ],
+  Stop: [
+    { hooks: [{ type: "command", command: cmd("session-end.ts") }] },
+  ],
+};
+
+for (const [event, entries] of Object.entries(betterdbHooks)) {
+  const prev = Array.isArray(existingHooks[event]) ? existingHooks[event] : [];
+  const filtered = prev.filter((entry) => {
+    const json = JSON.stringify(entry);
+    return !json.includes("betterdb") && !json.includes(hooksDir);
+  });
+  existingHooks[event] = [...filtered, ...entries];
+}
+settings["hooks"] = existingHooks;
+
+writeFileSync(settingsPath, JSON.stringify(settings, null, 2) + "\n");
+
+console.log("BetterDB Memory — Hooks registered in ~/.claude/settings.json\n");
+console.log("  SessionStart → session-start.ts");
+console.log("  PreToolUse   → pre-tool.ts");
+console.log("  PostToolUse  → post-tool.ts");
+console.log("  Stop         → session-end.ts");
+console.log(`\n  Plugin root: ${resolvedRoot}`);
+console.log("\n  Restart Claude Code for hooks to take effect.");

package/scripts/setup-index.ts

@@ -1,14 +1,21 @@
 #!/usr/bin/env bun
 import { getValkeyClient } from "../src/client/valkey.js";
+import { getPluginMemoryStore } from "../src/client/memory-store.js";
 import { createModelClient } from "../src/client/model.js";
-import { config } from "../src/config.js";
 
 const client = await getValkeyClient();
 const modelClient = await createModelClient();
 
-await client.ensureIndex(modelClient.embedDim, modelClient.preset.embedModel);
-console.log("Index ready:", config.valkey.indexName);
+// Create the episodic vector index that MemoryStore reads/writes
+// (betterdb:mem:idx) — the same one `install` builds. Record the active
+// provider/dimension first so a later provider swap is caught.
+await client.assertEmbedDim(modelClient.embedDim, modelClient.preset.embedModel);
+const store = await getPluginMemoryStore((t) => modelClient.embed(t));
+await store.ensureIndex();
+
+console.log("Index ready: betterdb:mem:idx");
 console.log("Embedding dimension:", modelClient.embedDim);
 console.log("Preset:", modelClient.preset.embedModel, "/", modelClient.preset.summarizeModel);
 
+await store.close();
 await client.quit();

package/scripts/unregister-hooks.ts

@@ -0,0 +1,79 @@
+#!/usr/bin/env bun
+
+/**
+ * Remove BetterDB Memory lifecycle hooks from ~/.claude/settings.json.
+ *
+ * Only removes hook entries whose commands contain "betterdb-memory" or
+ * "betterdb" in the path. Other hooks from other plugins are preserved.
+ *
+ * Usage:
+ *   bun run scripts/unregister-hooks.ts
+ */
+
+import { existsSync, readFileSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+
+const HOME = process.env["HOME"] ?? process.env["USERPROFILE"] ?? "";
+const settingsPath = join(HOME, ".claude", "settings.json");
+
+if (!existsSync(settingsPath)) {
+  console.log("No ~/.claude/settings.json found — nothing to remove.");
+  process.exit(0);
+}
+
+let settings: Record<string, unknown>;
+try {
+  settings = JSON.parse(readFileSync(settingsPath, "utf-8"));
+} catch {
+  console.error("ERROR: Could not parse ~/.claude/settings.json");
+  process.exit(1);
+}
+
+const hooks = settings["hooks"];
+if (!hooks || typeof hooks !== "object") {
+  console.log("No hooks found in ~/.claude/settings.json — nothing to remove.");
+  process.exit(0);
+}
+
+const hooksObj = hooks as Record<string, unknown[]>;
+const BETTERDB_PATTERN = /betterdb/i;
+let removedCount = 0;
+
+for (const [event, entries] of Object.entries(hooksObj)) {
+  if (!Array.isArray(entries)) continue;
+
+  const filtered = entries.filter((entry) => {
+    if (typeof entry !== "object" || entry === null) return true;
+    const hooksList = (entry as Record<string, unknown>)["hooks"];
+    if (!Array.isArray(hooksList)) return true;
+
+    // Keep this entry if ANY of its hooks are NOT betterdb-related
+    const hasBetterdb = hooksList.some((h) => {
+      if (typeof h !== "object" || h === null) return false;
+      const command = (h as Record<string, unknown>)["command"];
+      return typeof command === "string" && BETTERDB_PATTERN.test(command);
+    });
+
+    if (hasBetterdb) removedCount++;
+    return !hasBetterdb;
+  });
+
+  if (filtered.length === 0) {
+    delete hooksObj[event];
+  } else {
+    hooksObj[event] = filtered;
+  }
+}
+
+// If hooks object is now empty, remove it entirely
+if (Object.keys(hooksObj).length === 0) {
+  delete settings["hooks"];
+}
+
+writeFileSync(settingsPath, JSON.stringify(settings, null, 2) + "\n");
+
+if (removedCount > 0) {
+  console.log(`BetterDB Memory — Removed ${removedCount} hook(s) from ~/.claude/settings.json`);
+} else {
+  console.log("No BetterDB Memory hooks found in ~/.claude/settings.json.");
+}