alvin-bot 4.21.0 → 4.22.0

package/CHANGELOG.md

@@ -2,6 +2,67 @@
 
 All notable changes to Alvin Bot are documented here.
 
+## [4.22.0] — 2026-05-05
+
+### 🧠 Memory architecture overhaul: pluggable providers + smart inject
+
+Public users without `GOOGLE_API_KEY` (the v4.20–v4.21 default for embeddings) now get a working indexed memory store out of the box. The embeddings layer is refactored behind a provider interface with four backends auto-detected at startup:
+
+| Tier | Provider | Setup | Cost | Dim |
+|---|---|---|---|---|
+| 1 | Gemini (`gemini-embedding-001`) | `GOOGLE_API_KEY` | free tier | 3072 |
+| 2 | OpenAI (`text-embedding-3-small`) | `OPENAI_API_KEY` | ~$0.02 / 1M tokens | 1536 |
+| 3 | Ollama (default `nomic-embed-text`) | `ollama pull nomic-embed-text` | free, local, private | 768 |
+| 4 | **FTS5 (BM25 keyword)** | nothing | free | n/a |
+
+The FTS5 fallback is the headline: SQLite's built-in full-text-search virtual table with BM25 ranking. No API key, no network, no setup. Indexes the same chunks as the vector providers (`MEMORY.md`, daily logs, project files, hub memory, asset index) and ranks matches by relevance. Excellent for proper-noun and exact-term lookups (project names, commands, error messages); weaker than vector search for synonyms and conceptual paraphrase queries — but available everywhere.
+
+**Upgrade path.** A user starts on FTS5 (no keys needed). Later they set `GOOGLE_API_KEY` in their `.env` → next bot start detects the schema mismatch via `meta.embedding_model`, drops the FTS5 table, initialises the vector schema, and reindexes. Same in reverse. All seamless, no manual steps.
+
+Override the auto-detection with `EMBEDDINGS_PROVIDER=gemini|openai|ollama|fts5|auto` (default `auto`).
+
+### ✂️ MEMORY.md no longer bulk-injected into every system prompt (when SQLite is populated)
+
+Pre-v4.22, `MEMORY.md` (typically tens of KB of curated long-term knowledge) and the last two daily logs were plain-text-injected into the system prompt on **every turn**. With a populated SQLite store, the same content is available via the smaller, query-targeted `searchMemory()` retrieval — much smaller prompts, much more relevant context.
+
+New `MEMORY_INJECT_MODE` env var:
+
+- `auto` (default) — sqlite when the store has indexed entries, else legacy
+- `legacy` — pre-v4.22 behaviour, full plain-text inject every turn
+- `sqlite` — never plain-text-inject `MEMORY.md` or daily logs (force smart mode regardless of store state)
+
+Always plain-text injected regardless of mode: `identity.md` (L0) and `preferences.md` (L1) — these are tiny by design and contain always-on facts that semantic search may miss for short or generic queries. Recommended pattern: keep critical "never X" / "always Y" rules in `preferences.md`, let the bulk knowledge live in `MEMORY.md` and be retrieved on demand.
+
+For users still on the legacy monolithic `MEMORY.md` setup (no `identity.md`, no `preferences.md`), auto mode kicks in only after the SQLite store is populated — until then, plain-text injection of `MEMORY.md` continues to work as before. Zero-touch upgrade.
+
+### 🔇 Quieter logs for missing keys
+
+The `⚠️ Embeddings init failed: Google API key not configured` warning is gone — that startup line is now `ℹ️ Memory provider: fts5-bm25 (keyword-local). Initial index will run on first use.` Public users without Gemini no longer see a scary warning that suggested the bot was broken when in fact it was working correctly.
+
+### 🩺 `alvin-bot doctor` Memory section expanded
+
+Reports the active provider, dimension, indexed entry/file counts, last-reindex timestamp, and effective inject mode. For not-yet-initialised stores it predicts which provider will run on first start so users can confirm the auto-detection picked what they expected.
+
+```
+  Memory:
+  ✅ Provider: gemini-embedding-001 (vector-cloud, 3072-dim)
+     3827 entries / 316 files indexed, 48.8 MB on disk
+     Last reindex: 25 h ago
+     Inject mode: sqlite (auto)
+```
+
+### Architecture
+
+- New: `src/services/embeddings/` directory — `provider.ts` (interface), `vector-base.ts` (shared vector logic), `gemini.ts`, `openai.ts`, `ollama.ts`, `fts5.ts`, `auto-detect.ts`, `index.ts` (facade)
+- New: `src/services/memory-inject-mode.ts` — env resolver
+- Updated: `src/services/memory-layers.ts`, `src/services/memory.ts` — gate plain-text injection on inject mode
+- `src/services/embeddings.ts` is now a thin re-export shim — all existing imports keep working
+
+### Tests
+
+- 24 new tests across FTS5 provider, auto-detection, and inject-mode resolver
+- All 535 existing tests still pass (one pre-existing port-binding flake in `web-server-integration.test.ts` is unrelated)
+
 ## [4.21.0] — 2026-05-04
 
 ### 🌐 New skill: Agent Browser (Tier-1.5)

package/README.md

@@ -433,6 +433,13 @@ OPENROUTER_API_KEY=<key>        # OpenRouter (100+ models)
 PRIMARY_PROVIDER=claude-sdk     # Primary AI provider
 FALLBACK_PROVIDERS=nvidia-kimi-k2.5,nvidia-llama-3.3-70b
 
+# Memory backend (v4.22+) — auto-detects based on what keys you have.
+# Set to override the default priority: gemini → openai → ollama → fts5.
+# fts5 is the zero-config keyword fallback — no key needed, works for everyone.
+EMBEDDINGS_PROVIDER=auto                  # auto | gemini | openai | ollama | fts5
+OLLAMA_EMBEDDING_MODEL=nomic-embed-text   # only used for ollama provider
+MEMORY_INJECT_MODE=auto                   # auto | legacy | sqlite (see CHANGELOG v4.22)
+
 # Optional Platforms
 WHATSAPP_ENABLED=true           # Enable WhatsApp (needs Chrome)
 DISCORD_TOKEN=<token>           # Enable Discord

package/bin/cli.js

@@ -1408,13 +1408,12 @@ async function doctor() {
     console.log(`       npm i -g agent-browser && agent-browser install`);
   }
 
-  // ── Memory (semantic search backend) ──
+  // ── Memory (provider + index health) ──
   console.log("\n  Memory:");
   const embJson = resolve(DATA_DIR, "memory", ".embeddings.json");
   const embDb = resolve(DATA_DIR, "memory", ".embeddings.db");
   const embBakSqlite = resolve(DATA_DIR, "memory", ".embeddings.json.bak-pre-sqlite");
 
-  // better-sqlite3 native binary loadable?
   let sqliteOk = false;
   let sqliteErr = "";
   try {
@@ -1424,26 +1423,54 @@ async function doctor() {
   } catch (err) {
     sqliteErr = err instanceof Error ? err.message : String(err);
   }
-  if (sqliteOk) {
-    console.log(`  ✅ better-sqlite3 native binary loadable`);
-  } else {
-    console.log(`  ❌ better-sqlite3 native binary not loadable — semantic search disabled`);
+  if (!sqliteOk) {
+    console.log(`  ❌ better-sqlite3 native binary not loadable — memory store disabled`);
     console.log(`     Fix: cd $(npm root -g)/alvin-bot && npm rebuild better-sqlite3`);
     console.log(`     Detail: ${sqliteErr.split("\n")[0]}`);
-  }
-
-  if (sqliteOk && existsSync(embDb)) {
+  } else if (existsSync(embDb)) {
     try {
       const req = (await import("module")).createRequire(import.meta.url);
       const Database = req("better-sqlite3");
       const db = new Database(embDb, { readonly: true });
-      const entries = db.prepare("SELECT COUNT(*) AS c FROM entries").get().c;
-      const files = db.prepare("SELECT COUNT(*) AS c FROM file_mtimes").get().c;
-      const sizeMb = (statSync(embDb).size / 1024 / 1024).toFixed(0);
+      // Read provider + meta
+      let model = "unknown", tier = "unknown", dim = 0, lastReindex = 0;
+      try {
+        const meta = db.prepare("SELECT key, value FROM meta").all();
+        const m = Object.fromEntries(meta.map(r => [r.key, r.value]));
+        // v4.22 keys preferred; fall back to v4.20 legacy "model" key.
+        // Legacy v4.20 DBs only have meta.model (always Gemini-format). v4.22+
+        // sets meta.embedding_model with a tier-prefixed name.
+        model = m.embedding_model || m.model || "unknown";
+        tier = m.embedding_tier || (m.model ? "vector-cloud" : "unknown");
+        dim = Number(m.embedding_dim || 0);
+        lastReindex = Number(m.lastReindex || 0);
+      } catch { /* meta table missing */ }
+
+      // Count rows in whichever provider table exists.
+      let entries = 0;
+      for (const tbl of ["entries", "entries_fts"]) {
+        try {
+          entries = db.prepare(`SELECT COUNT(*) AS c FROM ${tbl}`).get().c;
+          if (entries > 0) break;
+        } catch { /* table missing */ }
+      }
+      const files = (() => {
+        try { return db.prepare("SELECT COUNT(*) AS c FROM file_mtimes").get().c; } catch { return 0; }
+      })();
+      const sizeMb = (statSync(embDb).size / 1024 / 1024).toFixed(1);
       db.close();
-      console.log(`  ✅ Vector store: ${entries} entries across ${files} sources (${sizeMb} MB SQLite)`);
+
+      console.log(`  ✅ Provider: ${model}${dim ? ` (${tier}, ${dim}-dim)` : ` (${tier})`}`);
+      console.log(`     ${entries} entries / ${files} files indexed, ${sizeMb} MB on disk`);
+      if (lastReindex) {
+        const ago = Math.round((Date.now() - lastReindex) / 1000 / 60);
+        console.log(`     Last reindex: ${ago < 60 ? `${ago} min ago` : `${Math.round(ago / 60)} h ago`}`);
+      }
+      const injectMode = (getEnv("MEMORY_INJECT_MODE") || "auto").toLowerCase();
+      const effective = injectMode === "auto" ? (entries > 0 ? "sqlite" : "legacy") : injectMode;
+      console.log(`     Inject mode: ${effective}${injectMode === "auto" ? " (auto)" : ""}`);
     } catch (err) {
-      console.log(`  ⚠️  Vector store exists but unreadable: ${err.message}`);
+      console.log(`  ⚠️  Memory store exists but unreadable: ${err.message}`);
     }
   } else if (existsSync(embJson)) {
     const sizeMb = (statSync(embJson).size / 1024 / 1024).toFixed(0);
@@ -1451,7 +1478,13 @@ async function doctor() {
   } else if (existsSync(embBakSqlite)) {
     console.log(`  ✅ Migration to SQLite already done (legacy JSON kept as .bak-pre-sqlite)`);
   } else {
-    console.log(`  ℹ️  No vector store yet — will be built on first message`);
+    // Predict which provider will be picked on first start.
+    const hasGoogle = !!getEnv("GOOGLE_API_KEY");
+    const hasOpenAI = !!getEnv("OPENAI_API_KEY");
+    console.log(`  ℹ️  Memory store not initialised yet (will be on first bot start)`);
+    if (hasGoogle) console.log(`     Will use: Gemini (3072-dim, semantic)`);
+    else if (hasOpenAI) console.log(`     Will use: OpenAI text-embedding-3-small (1536-dim, semantic)`);
+    else console.log(`     Will use: FTS5 keyword (zero-config). Set GOOGLE_API_KEY or OPENAI_API_KEY for semantic vectors.`);
   }
 
   // ── Extras ──

package/dist/handlers/commands.js

@@ -370,7 +370,12 @@ export function registerCommands(bot) {
         // Memory stats
         const memStats = getMemoryStats();
         const idxStats = getIndexStats();
-        const memLine = `${memStats.dailyLogs} days, ${memStats.todayEntries} entries today, ${formatBytes(memStats.longTermSize)} LTM | 🔍 ${idxStats.entries} vectors`;
+        const { getEffectiveInjectMode, getInjectModeRaw } = await import("../services/memory-inject-mode.js");
+        const injectMode = getEffectiveInjectMode();
+        const injectRaw = getInjectModeRaw();
+        const indexLabel = idxStats.tier === "keyword-local" ? "FTS5" : "vec";
+        const modeLabel = injectRaw === "auto" ? `${injectMode}(auto)` : injectMode;
+        const memLine = `${memStats.dailyLogs} days, ${memStats.todayEntries} entries today, ${formatBytes(memStats.longTermSize)} LTM | 🔍 ${idxStats.entries} ${indexLabel} (${idxStats.provider}) | inject:${modeLabel}`;
         // Provider health + failover state
         const healthRows = getHealthStatus();
         const failedOver = isFailedOver();

package/dist/services/embeddings/auto-detect.js

@@ -0,0 +1,74 @@
+/**
+ * Provider auto-detection for the memory backend.
+ *
+ * Probes available providers in priority order and returns the first one that
+ * is usable right now. The order is:
+ *
+ *   1. EMBEDDINGS_PROVIDER env override (gemini|openai|ollama|fts5) — explicit wins.
+ *   2. Gemini  (free tier, 3072-dim) — when GOOGLE_API_KEY is set.
+ *   3. OpenAI  (cheap, 1536-dim)     — when OPENAI_API_KEY is set.
+ *   4. Ollama  (local, free, 768-dim default) — when /api/tags responds AND
+ *               an embedding model is pulled. Many Ollama users only have chat
+ *               models, so we don't auto-pull; we return false from isAvailable.
+ *   5. FTS5    (always available)   — universal zero-config fallback.
+ *
+ * The facade calls this once per startup and caches the chosen provider for
+ * the lifetime of the process. If the user changes EMBEDDINGS_PROVIDER or
+ * adds a key, a restart picks up the new choice (and triggers a reindex via
+ * schema-mismatch detection in the facade).
+ */
+import { GeminiProvider } from "./gemini.js";
+import { OpenAIProvider } from "./openai.js";
+import { OllamaProvider } from "./ollama.js";
+import { Fts5Provider } from "./fts5.js";
+export function parseProviderKey(raw) {
+    const v = (raw ?? "").trim().toLowerCase();
+    switch (v) {
+        case "gemini":
+        case "openai":
+        case "ollama":
+        case "fts5":
+        case "auto":
+            return v;
+        default:
+            return "auto";
+    }
+}
+function instantiate(key) {
+    switch (key) {
+        case "gemini":
+            return new GeminiProvider();
+        case "openai":
+            return new OpenAIProvider();
+        case "ollama":
+            return new OllamaProvider();
+        case "fts5":
+            return new Fts5Provider();
+    }
+}
+/**
+ * Pick the active provider. If override is given (and not "auto"), force it
+ * regardless of availability — the facade still runs isAvailable() and
+ * surfaces a clear error if the forced provider can't actually run.
+ *
+ * Otherwise probe in priority order until one succeeds. FTS5 is the universal
+ * tail and always succeeds (assuming better-sqlite3 loaded).
+ */
+export async function detectProvider(override) {
+    if (override && override !== "auto") {
+        return instantiate(override);
+    }
+    const tryOrder = ["gemini", "openai", "ollama", "fts5"];
+    for (const key of tryOrder) {
+        const p = instantiate(key);
+        try {
+            if (await p.isAvailable())
+                return p;
+        }
+        catch {
+            // probe failure is non-fatal — try next
+        }
+    }
+    // unreachable: fts5.isAvailable always returns true
+    return new Fts5Provider();
+}

package/dist/services/embeddings/fts5.js

@@ -0,0 +1,108 @@
+/**
+ * FTS5 Memory Provider — zero-config keyword search via SQLite full-text.
+ *
+ * No API keys, no network, no embeddings. Indexes chunk text into an FTS5
+ * virtual table and ranks matches via BM25. Universal fallback when the user
+ * has no Gemini / OpenAI / Ollama configured. Excellent for proper-noun and
+ * exact-term lookups (project names, commands, error messages); weaker than
+ * vector search for synonyms and conceptual paraphrase queries.
+ *
+ * Schema:
+ *   entries_fts (id UNINDEXED, source UNINDEXED, text)
+ *   tokenizer: unicode61 with diacritic stripping (works for de/en mixed memory).
+ *
+ * Score normalisation: SQLite's bm25() returns negative numbers (more negative
+ * = more relevant). We map to [0, 1] via 1 / (1 + |bm25|) so callers can use
+ * the same minScore semantics as vector providers.
+ */
+const TABLE = "entries_fts";
+/** FTS5 has reserved characters/operators in MATCH queries. Sanitize to plain
+ *  word-OR by extracting alphanumeric tokens and quoting each as a phrase. */
+function sanitizeQuery(query) {
+    const tokens = query
+        .toLowerCase()
+        .split(/[\s\W]+/u)
+        .filter(t => t.length >= 2 && t.length <= 64);
+    if (tokens.length === 0)
+        return "";
+    // Each token wrapped in double quotes makes it a literal phrase, immune to
+    // FTS5 operator characters (NEAR, AND, OR, NOT, *, etc.). Joined with OR.
+    return tokens.map(t => `"${t.replace(/"/g, '""')}"`).join(" OR ");
+}
+export class Fts5Provider {
+    name = "fts5-bm25";
+    dim = 0;
+    tier = "keyword-local";
+    async isAvailable() {
+        return true;
+    }
+    initSchema(db) {
+        // FTS5 doesn't allow secondary indexes on the virtual table itself;
+        // source filtering happens via WHERE clauses on the UNINDEXED column,
+        // which is fast enough at our corpus size (<100k chunks).
+        db.exec(`
+      CREATE VIRTUAL TABLE IF NOT EXISTS ${TABLE} USING fts5(
+        id UNINDEXED,
+        source UNINDEXED,
+        text,
+        tokenize = 'unicode61 remove_diacritics 2'
+      );
+    `);
+    }
+    dropSchema(db) {
+        db.exec(`DROP TABLE IF EXISTS ${TABLE};`);
+    }
+    async indexChunks(db, chunks) {
+        if (chunks.length === 0)
+            return;
+        const ins = db.prepare(`INSERT INTO ${TABLE} (id, source, text) VALUES (?, ?, ?)`);
+        const writeAll = db.transaction((rows) => {
+            for (const c of rows)
+                ins.run(c.id, c.source, c.text);
+        });
+        writeAll(chunks);
+    }
+    dropEntriesForSources(db, sources) {
+        if (sources.length === 0)
+            return;
+        const del = db.prepare(`DELETE FROM ${TABLE} WHERE source = ?`);
+        const dropAll = db.transaction((srcs) => {
+            for (const s of srcs)
+                del.run(s);
+        });
+        dropAll(sources);
+    }
+    async search(db, query, topK, minScore) {
+        const matchExpr = sanitizeQuery(query);
+        if (!matchExpr)
+            return [];
+        let rows;
+        try {
+            rows = db
+                .prepare(`SELECT source, text, bm25(${TABLE}) AS bm25 FROM ${TABLE} WHERE ${TABLE} MATCH ? ORDER BY bm25(${TABLE}) LIMIT ?`)
+                .all(matchExpr, topK * 3);
+        }
+        catch {
+            // FTS5 MATCH parse errors (e.g. exotic Unicode) → return empty.
+            return [];
+        }
+        const results = rows
+            .map(r => ({
+            text: r.text,
+            source: r.source,
+            score: 1 / (1 + Math.abs(r.bm25)),
+        }))
+            .filter(r => r.score >= minScore)
+            .slice(0, topK);
+        return results;
+    }
+    countEntries(db) {
+        try {
+            const row = db.prepare(`SELECT COUNT(*) AS c FROM ${TABLE}`).get();
+            return row?.c ?? 0;
+        }
+        catch {
+            return 0;
+        }
+    }
+}

package/dist/services/embeddings/gemini.js

@@ -0,0 +1,65 @@
+/**
+ * Gemini Memory Provider — Google's gemini-embedding-001 (3072-dim).
+ *
+ * Uses the public Generative Language API. Free tier limits: 100 RPM, 30k TPM,
+ * 1500 RPD as of 2026-04. Batches up to 100 texts per request via
+ * batchEmbedContents. RETRIEVAL_DOCUMENT for index, RETRIEVAL_QUERY for search.
+ */
+import { config } from "../../config.js";
+import { VectorProviderBase } from "./vector-base.js";
+const MODEL = "gemini-embedding-001";
+const BATCH_SIZE = 100;
+export class GeminiProvider extends VectorProviderBase {
+    name = MODEL;
+    dim = 3072;
+    tier = "vector-cloud";
+    async isAvailable() {
+        return Boolean(config.apiKeys.google);
+    }
+    async embed(texts) {
+        const apiKey = config.apiKeys.google;
+        if (!apiKey)
+            throw new Error("GOOGLE_API_KEY not configured");
+        const out = [];
+        for (let i = 0; i < texts.length; i += BATCH_SIZE) {
+            const batch = texts.slice(i, i + BATCH_SIZE);
+            const res = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${MODEL}:batchEmbedContents?key=${apiKey}`, {
+                method: "POST",
+                headers: { "Content-Type": "application/json" },
+                body: JSON.stringify({
+                    requests: batch.map(text => ({
+                        model: `models/${MODEL}`,
+                        content: { parts: [{ text }] },
+                        taskType: "RETRIEVAL_DOCUMENT",
+                    })),
+                }),
+            });
+            if (!res.ok) {
+                throw new Error(`Gemini embeddings API error: ${res.status} — ${await res.text()}`);
+            }
+            const data = (await res.json());
+            for (const e of data.embeddings)
+                out.push(e.values);
+        }
+        return out;
+    }
+    async embedQuery(text) {
+        const apiKey = config.apiKeys.google;
+        if (!apiKey)
+            throw new Error("GOOGLE_API_KEY not configured");
+        const res = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${MODEL}:embedContent?key=${apiKey}`, {
+            method: "POST",
+            headers: { "Content-Type": "application/json" },
+            body: JSON.stringify({
+                model: `models/${MODEL}`,
+                content: { parts: [{ text }] },
+                taskType: "RETRIEVAL_QUERY",
+            }),
+        });
+        if (!res.ok) {
+            throw new Error(`Gemini embeddings API error: ${res.status} — ${await res.text()}`);
+        }
+        const data = (await res.json());
+        return data.embedding.values;
+    }
+}